穆再排尔·穆合塔尔

R Research / NeurIPS 2026

Extended abstract · 6 pp.

Selective Cross-Modal Alignment in Vision-Language Models: Why Position Grounding Succeeds but Physical Reasoning Fails

视觉语言模型中的选择性跨模态对齐:位置接地为何成功,物理推理为何失败

Author

Muzapar Muhtar — independent author

Venue

NeurIPS 2026 Workshop · NeurReps · Extended Abstract

Pages

6 pp. · PDF

Abstract

EN

Physical intelligence—the ability to perceive dynamics and anticipate what happens next—is central to general intelligence, yet little is known about whether vision–language models (VLMs) internally represent it. We probe Qwen2.5-VL-3B on 6,000 fully controlled synthetic physics clips and uncover a selective cross-modal alignment failure: position is linearly decodable from visual tokens (R² = 0.97) and perfectly grounded in language (100% success), whereas velocity is encoded only as an implicit, entangled manifold (R² = 0.42, non-linear probe) that language generation does not use (19% physical-reasoning accuracy; less than 5% causal effect when the velocity-encoding layers are patched). This is not a missing capability: prompt engineering raises physical reasoning by +62% and eliminates a systematic response bias, while leaving the visual representations unchanged (cosine similarity ≈1.0). Comparisons with DINOv2 and SigLIP show that language training created the velocity encoding (R²: 0.04/0.07 → 0.42), yet the default readout ignores it. Cross-modal alignment in VLMs is therefore selective and prompt-dependent: a model can hold physical knowledge it does not use, and capability must be measured by probing and activation, not by behavior alone.

  • vision-language models
  • representation geometry
  • probing
  • cross-modal alignment
  • physical reasoning

At a glance

R² 0.97 position / linear probe
100% language grounding
R² 0.42 velocity / non-linear probe
19% physical reasoning
+62% with prompting
01

Position is explicit. Velocity is a manifold.

Position is linearly decodable at R² = 0.97; velocity reaches only R² = 0.34 with a linear probe and 0.42 with a non-linear one, below the constant-velocity baseline. Direction is spread across nearly all 1,280 dimensions.

02

Language does not use it by default.

The model grounds position with 100% success but predicts motion with only 19%. Patching the velocity layers changes accuracy by 4.5%. Prompting raises accuracy from 21% to 34% (+62%) and removes the 73% “C” bias.

03

Language training creates the encoding.

DINOv2 and SigLIP reach R² = 0.04 and 0.07 on velocity; Qwen2.5-VL reaches 0.42. The representation is learned, but the default readout still ignores it.

FIG. 1 / LAYER-WISE PROBINGQWEN2.5-VL-3B0.00.20.40.60.81.0051015202530LAYER INDEXR² SCORE0.970.420.15● POSITION / LINEAR■ VELOCITY / NON-LINEAR--- CONSTANT VELOCITY 0.90··· RANDOM 0.25
FIG. 1Layer-wise probing. Position decodes near-perfectly from mid layers; velocity peaks far lower, below the constant-velocity baseline. Redrawn from the paper.
FIG. 2 / GROUNDING VS PHYSICAL REASONING100 PAIRED QUERIES100%GROUNDING(POSITION)19%PHYSICAL REASONING(VELOCITY)81%GAPUNDERSTANDS BUT CANNOT PREDICTSELECTIVE CROSS-MODAL ALIGNMENT / QWEN2.5-VL-3B
FIG. 2Grounding vs. physical reasoning on 100 paired queries. 81% of samples understand position but cannot predict motion.
FIG. 3 / PROMPT ENGINEERING6 FORMULATIONS / 100 CLIPS EACHACCURACYPHYSICAL0.34VISUAL GROUNDING0.34ANTI-BIAS0.31EXPLICIT0.30COT0.29BASELINE0.21BASELINE 0.21PROMPTING RAISES ACCURACY +62%C-OPTION BIAS0%0%0%0%0%73%PROMPTING ELIMINATES THE BIAS
FIG. 3Six prompt formulations raise accuracy from 21% to 34% and remove the 73% “C” bias.
First page of the extended abstract

PDF / 6 pp.

Read the full extended abstract

The PDF is the submitted extended abstract. Six pages, two-column, with the original figures and references.

Download PDF

Citation

Muhtar, M. (2026). Selective Cross-Modal Alignment in Vision-Language Models: Why Position Grounding Succeeds but Physical Reasoning Fails. NeurIPS 2026 Workshop (NeurReps), Extended Abstract.