02 2026 / NeurIPS 2026 Workshop · NeurReps · extended abstract
Selective Alignment
What Qwen2.5-VL can decode, and what it refuses to reason about.
A vision-language model can describe a video with unsettling fluency. It can say that a ball is rolling down a ramp, that a cup is falling, that a person is walking. The question is narrower: which of those properties are actually represented, and which are only inferred from language?
The question
The paper studies selective cross-modal alignment: the possibility that a model grounds one property of the visual world well while failing to ground another, in the same representation, on the same data.
I chose physical motion because it has separable properties. Position is spatial and static. Velocity is relational and temporal. If a model can talk about both but only represents one, the difference should be visible in a linear probe.
The setup
Qwen2.5-VL-3B, 6,000 controlled videos. Each clip varies one property at a time: position, velocity, direction, collision. For each clip we extract hidden states, train simple linear probes, and compare probe accuracy with the model's answers to direct questions.
The controls matter. Synthetic videos are not a substitute for the world, but they let us hold the language constant while changing the visual variable. If the model's description changes when the pixels change, the description is grounded in something.
What the probe found
Position is linearly decodable with an R² of 0.97. Language grounding is 100%: the model names the location correctly across the dataset, in the right frame.
Velocity is not. The information is present — probes beat chance — but it is entangled. There is no clean direction for speed. The geometry is a manifold, not an axis. When asked directly, the model often falls back to a linguistic prior: things fall, things slow down, things speed up.
Prompting
A carefully worded prompt improves physical reasoning by 62%. The number is real, and it is a warning. It suggests the information is not absent; it is inaccessible to the model's default procedure. A prompt is a procedure supplied from outside. It helps without changing what the model represents.
This is why the paper reports the prompted result separately. "The model can reason about physics" and "the model can be prompted to produce a better answer about physics" are different claims.
What it means
The result is narrow: one model family, 3B parameters, synthetic video, controlled variables. It does not claim that larger models fail in the same way. It claims that grounding is selective, and it shows a method for locating the selectivity.
The next experiment is obvious: can the representation be trained to separate velocity without losing position? I do not know yet. That is why it is next.
The model had learned to say the right kind of thing about motion without building the thing that motion requires.
Measurements