穆再排尔·穆合塔尔
← All work

02 2026 / NeurIPS 2026 Workshop · NeurReps · extended abstract

Selective Alignment

What Qwen2.5-VL can decode, and what it refuses to reason about.

Role
Independent author
Period
2026
Stack
Qwen2.5-VL-3B · PyTorch · linear probes · controlled video
Status
NeurIPS 2026 Workshop OpenReview GitHub
Fine black lines on paper forming a moiré interference pattern, cut by one vermilion line.
PL. 02 / Selective Alignment Fine black lines on paper forming a moiré interference pattern, cut by one vermilion line. 2026

A vision-language model can describe a video with unsettling fluency. It can say that a ball is rolling down a ramp, that a cup is falling, that a person is walking. The question is narrower: which of those properties are actually represented, and which are only inferred from language?

The question

The paper studies selective cross-modal alignment: the possibility that a model grounds one property of the visual world well while failing to ground another, in the same representation, on the same data.

I chose physical motion because it has separable properties. Position is spatial and static. Velocity is relational and temporal. If a model can talk about both but only represents one, the difference should be visible in a linear probe.

The setup

Qwen2.5-VL-3B, 6,000 controlled videos. Each clip varies one property at a time: position, velocity, direction, collision. For each clip we extract hidden states, train simple linear probes, and compare probe accuracy with the model's answers to direct questions.

The controls matter. Synthetic videos are not a substitute for the world, but they let us hold the language constant while changing the visual variable. If the model's description changes when the pixels change, the description is grounded in something.

What the probe found

Position is linearly decodable with an R² of 0.97. Language grounding is 100%: the model names the location correctly across the dataset, in the right frame.

Velocity is not. The information is present — probes beat chance — but it is entangled. There is no clean direction for speed. The geometry is a manifold, not an axis. When asked directly, the model often falls back to a linguistic prior: things fall, things slow down, things speed up.

Prompting

A carefully worded prompt improves physical reasoning by 62%. The number is real, and it is a warning. It suggests the information is not absent; it is inaccessible to the model's default procedure. A prompt is a procedure supplied from outside. It helps without changing what the model represents.

This is why the paper reports the prompted result separately. "The model can reason about physics" and "the model can be prompted to produce a better answer about physics" are different claims.

What it means

The result is narrow: one model family, 3B parameters, synthetic video, controlled variables. It does not claim that larger models fail in the same way. It claims that grounding is selective, and it shows a method for locating the selectivity.

The next experiment is obvious: can the representation be trained to separate velocity without losing position? I do not know yet. That is why it is next.

The model had learned to say the right kind of thing about motion without building the thing that motion requires.

Measurements

R² 0.97 position decodable
100% language grounding
+62% with prompting
6,000 controlled videos

Next — 03 / 2026

Nuclear Boy

An AI coding agent that lives on an Android phone.