R Research / NeurIPS 2026
扩展摘要 · 6 pp.
Selective Cross-Modal Alignment in Vision-Language Models: Why Position Grounding Succeeds but Physical Reasoning Fails
视觉语言模型中的选择性跨模态对齐:位置接地为何成功,物理推理为何失败
摘要
EN
物理智能——感知动态并预判下一步的能力——是通用智能的核心,但视觉语言模型是否在内部表征它,仍然缺少答案。我们在 6,000 条完全可控的合成物理片段上探测 Qwen2.5-VL-3B,发现了一种“选择性跨模态对齐失败”:位置可以从视觉 token 中线性解码(R² = 0.97),并在语言中完美接地(100%);速度却只以隐式、纠缠的流形存在(R² = 0.42,非线性探针),语言生成并不使用它(物理推理准确率 19%;对速度编码层做干预的因果效应低于 5%)。这不是能力缺失:提示工程把物理推理提升了 62%,并消除了系统性的回答偏差,同时不改变视觉表征(余弦相似度约 1.0)。与 DINOv2、SigLIP 的对照说明,速度编码是语言训练“创造”出来的(R²:0.04/0.07 → 0.42),但默认读出路径忽略了它。视觉语言模型中的跨模态对齐因此是选择性的、依赖提示的:模型可以持有它不使用的物理知识;能力必须通过探测与激活来测量,而不能只看行为。
Read the English abstract ↗
Physical intelligence—the ability to perceive dynamics and anticipate what happens next—is central to general intelligence, yet little is known about whether vision–language models (VLMs) internally represent it. We probe Qwen2.5-VL-3B on 6,000 fully controlled synthetic physics clips and uncover a selective cross-modal alignment failure: position is linearly decodable from visual tokens (R² = 0.97) and perfectly grounded in language (100% success), whereas velocity is encoded only as an implicit, entangled manifold (R² = 0.42, non-linear probe) that language generation does not use (19% physical-reasoning accuracy; less than 5% causal effect when the velocity-encoding layers are patched). This is not a missing capability: prompt engineering raises physical reasoning by +62% and eliminates a systematic response bias, while leaving the visual representations unchanged (cosine similarity ≈1.0). Comparisons with DINOv2 and SigLIP show that language training created the velocity encoding (R²: 0.04/0.07 → 0.42), yet the default readout ignores it. Cross-modal alignment in VLMs is therefore selective and prompt-dependent: a model can hold physical knowledge it does not use, and capability must be measured by probing and activation, not by behavior alone.
- vision-language models
- representation geometry
- probing
- cross-modal alignment
- physical reasoning
一览
位置是显式的,速度是流形。
位置可以线性解码,R² = 0.97;速度在线性探针下只有 0.34,非线性探针也只有 0.42,低于匀速外推基线。运动方向被分散在近 1,280 个维度里。
语言默认不使用它。
模型对位置的接地成功率是 100%,对运动的预测只有 19%。干预速度层只让准确率变化 4.5%。提示工程把准确率从 21% 提到 34%(+62%),并消除了 73% 的“C”偏差。
语言训练创造了这种编码。
DINOv2 与 SigLIP 的速度 R² 分别是 0.04 和 0.07,Qwen2.5-VL 达到 0.42。表征确实被学出来了,但默认读出路径仍然忽略它。
引用
Muhtar, M. (2026). Selective Cross-Modal Alignment in Vision-Language Models: Why Position Grounding Succeeds but Physical Reasoning Fails. NeurIPS 2026 Workshop (NeurReps), Extended Abstract.