Research / 20 Sep 2026
Position Is Decodable. Physics Is Not.
What a 3B vision-language model knows about motion, and where the knowledge stops.
A vision-language model can describe a video with unsettling fluency. It can say that a ball is rolling down a ramp, that a cup is falling, that a person is walking. The question I wanted to answer was narrower: which of those properties are actually represented in the model, and which are only inferred from the language it has read?
The paper studies Qwen2.5-VL-3B on 6,000 controlled physical videos. The setup is deliberately plain. Each clip varies one physical property at a time: position, velocity, direction, collision. We probe the model's internal representations with simple linear classifiers, and we compare those probes to what the model says when asked directly.
What we found
Position is linearly decodable. A probe can recover an object's spatial location from the model's hidden states with an R² of 0.97, and the model's language grounding is essentially perfect: it can name where the object is, in the right coordinate frame, across the dataset.
Velocity is not. The information is present — a probe can do better than chance — but it is entangled. There is no clean, low-dimensional direction that corresponds to speed. The geometry is a manifold, not an axis. When the model is asked to reason about speed, it often reaches for the linguistic prior: things fall, things slow down, things speed up. The prior is usually right about the sentence and sometimes wrong about the clip.
Why prompting helps
A carefully worded prompt improves physical reasoning by 62%. That number is real, and it is also a warning. The improvement suggests that the relevant information is not absent; it is inaccessible to the model's default procedure. A prompt is a procedure supplied from outside. It helps, but it does not change what the model has learned to represent.
This distinction matters for how we report results. "The model can reason about physics" and "the model can be prompted to produce a better answer about physics" are different claims. The second one is what we measured.
Where it stops
The study has clear limits. It looks at one model family, at 3B parameters, on synthetic videos with controlled variables. It does not claim that larger models fail in the same way, or that the manifold is universal. The contribution is a method and a negative result: a demonstration that grounding can succeed for one property while failing for another, in the same model, on the same data.
That is the kind of result I trust. It does not say that vision-language models cannot understand physics. It says that this one does not, in the way the benchmark assumed, and it shows where the assumption breaks.
The next experiment is the obvious one: can we train the representation to separate velocity without losing the position grounding? I do not know yet. That is why it is the next experiment.