R Research / NeurIPS 2026
Extended abstract · 6 pp.
Selective Cross-Modal Alignment in Vision-Language Models: Why Position Grounding Succeeds but Physical Reasoning Fails
视觉语言模型中的选择性跨模态对齐:位置接地为何成功,物理推理为何失败
Abstract
EN
Physical intelligence—the ability to perceive dynamics and anticipate what happens next—is central to general intelligence, yet little is known about whether vision–language models (VLMs) internally represent it. We probe Qwen2.5-VL-3B on 6,000 fully controlled synthetic physics clips and uncover a selective cross-modal alignment failure: position is linearly decodable from visual tokens (R² = 0.97) and perfectly grounded in language (100% success), whereas velocity is encoded only as an implicit, entangled manifold (R² = 0.42, non-linear probe) that language generation does not use (19% physical-reasoning accuracy; less than 5% causal effect when the velocity-encoding layers are patched). This is not a missing capability: prompt engineering raises physical reasoning by +62% and eliminates a systematic response bias, while leaving the visual representations unchanged (cosine similarity ≈1.0). Comparisons with DINOv2 and SigLIP show that language training created the velocity encoding (R²: 0.04/0.07 → 0.42), yet the default readout ignores it. Cross-modal alignment in VLMs is therefore selective and prompt-dependent: a model can hold physical knowledge it does not use, and capability must be measured by probing and activation, not by behavior alone.
- vision-language models
- representation geometry
- probing
- cross-modal alignment
- physical reasoning
At a glance
Position is explicit. Velocity is a manifold.
Position is linearly decodable at R² = 0.97; velocity reaches only R² = 0.34 with a linear probe and 0.42 with a non-linear one, below the constant-velocity baseline. Direction is spread across nearly all 1,280 dimensions.
Language does not use it by default.
The model grounds position with 100% success but predicts motion with only 19%. Patching the velocity layers changes accuracy by 4.5%. Prompting raises accuracy from 21% to 34% (+62%) and removes the 73% “C” bias.
Language training creates the encoding.
DINOv2 and SigLIP reach R² = 0.04 and 0.07 on velocity; Qwen2.5-VL reaches 0.42. The representation is learned, but the default readout still ignores it.
PDF / 6 pp.
Read the full extended abstract
The PDF is the submitted extended abstract. Six pages, two-column, with the original figures and references.
Download PDFCitation
Muhtar, M. (2026). Selective Cross-Modal Alignment in Vision-Language Models: Why Position Grounding Succeeds but Physical Reasoning Fails. NeurIPS 2026 Workshop (NeurReps), Extended Abstract.