穆再排尔·穆合塔尔
← Field notes

Data / 14 Mar 2026

The 18.4% Problem

367 bounding boxes were flipped. The model was not the problem.

2 min read Published 14 Mar 2026 Nanjing, China

The dataset had 1,999 frames. The detector was learning something. The validation loss went down, the boxes looked plausible, and the mAP was respectable. Then we visualized the predictions and noticed a pattern: a suspicious number of boxes were rotated 180 degrees.

Not wrong in the usual way. Not loose, not clipped, not confused by occlusion. Flipped. The kind of mistake a person makes when they are labeling quickly and the object has a symmetry that invites it.

The audit

We wrote a small tool to check the annotations before training. It did three things.

First, it checked geometric consistency. For each box, it compared the annotated corners to the expected orientation of the object class. A landing marker has a canonical orientation; if the box was rotated, the tool flagged it.

Second, it checked agreement between near-duplicate frames. Consecutive frames in a video should have similar boxes. A sudden flip in the middle of a sequence is more likely to be a label error than a physical event.

Third, it sampled frames for human review. Not random frames: the ones where the checks disagreed, where the model was confident and wrong, and where the class distribution was unusual. The tool did not replace judgment. It pointed it at the right places.

The audit found 367 flipped boxes out of 1,999 — 18.4%.

What the model learned

The model had learned the flip. It was not making an independent error; it was reproducing a consistent label. This is the quiet danger of a clean training pipeline: a systematic annotation mistake becomes a systematic prediction, and the loss curve looks healthy the whole time.

When we corrected the labels and retrained, the predictions became more stable across frames. The mAP improved, but the more important change was qualitative: the model stopped treating the marker as an ambiguous object and started treating it as an object with an orientation.

The fix

The fix was not only a corrected dataset. It was a check that runs before every training job, and a rule that no dataset version ships without an audit report. The report is boring: counts, examples, disagreement rates. Boring is the point. If the report is empty, we know the audit ran; if it is not, we know where to look.

The lesson

A model does not fail loudly. It fails in the space between the label and the thing itself. The 18.4% was not a model failure. It was a measurement failure that the model faithfully inherited.

I keep the number in the site's statement because it is the cleanest example I have of why the unglamorous parts of vision — annotation, evaluation, versioning — deserve the same attention as architecture. The network is the easy part to show. The label is the part that decides what the network becomes.