Wrong Answer, Different Wrong Answer

Vision-language models are being pointed at video to do action tracking: watch a clip, report what happened and who did it. I built a test to check whether one of them was really watching, and the test came back positive. Then I broke the number into its parts and found it had been measuring nothing at all.

1. What was I testing?

Point a vision-language model at six seconds of a pick-and-roll and ask what happened. You get back something like this:

“A player with jersey number 23 and jersey_color white made a 3PT Shot which result was made and was assisted by other player with jersey number 7.”

That is a structured event, pulled straight out of raw footage, with no bespoke tracking pipeline behind it. If it’s reliable, it’s genuinely useful: you could tag a season of film with it. It reads like a scout wrote it.

The trouble is that broadcast footage is full of things that correlate with the right answer without being the right answer. The home team wears white, so team identity is a wash of pixels. The scoreboard sits burned into the same corner of every frame. Highlight clips get cut around exactly one scoring event, so “something scored” is a safe bet before you look at a single pixel.

A model can score well on all of that without ever tracking a person. Which is a problem, because tracking the person is the entire reason you wanted the model.

So I built a harness to test it. Take away a cue that shouldn’t matter, see what the model does, and use that to separate real understanding from correlation. That’s a standard approach and I had no reason to doubt it.

What I found is that the standard approach can rank the cheating backwards. That turned out to be the more interesting result, so it’s most of what follows.

2. How do these models get built?

The failure I'm going to describe is much easier to accept once you know how the model was put together, because nothing in the assembly ever asked it to do the thing it can't do. Four stages, one at a time.

Where do the visual features come from?

The first stage is contrastive pretraining, the recipe CLIP made popular. You take a big batch of images and their captions, and you train two encoders so that each image lands near its own caption in a shared space. The training signal is a grid of similarity scores.

This gives you an encoder with a great sense of what's in a scene. But notice what it never has to do. The things it's pushing away are other images in the batch: a beach, a dog, a kitchen. So the pressure is to tell a basketball court apart from a kitchen. There's almost no pressure to tell one arrangement of players apart from another one.

Hold onto that. It comes back.

How do you bolt that onto a language model?

People tried a few designs here, including gated cross-attention and learned query bottlenecks. The one that won on sheer simplicity was: take the patch embeddings the vision encoder produces, run them through a small network that maps them into the language model's word-embedding space, and then let the decoder treat them as if they were words.

In Qwen2.5-VL the merge and the projection are one operation. Four patches from a 2x2 block get glued into a 5,120-long vector, then a small MLP squeezes that into one 2,048-dim token. The vision encoder works in 1,280 dims across 32 layers; the decoder works in 2,048.

That little MLP is the entire interface between seeing and speaking. It's worth sitting with how thin that is.

What does the decoder actually see?

This is the bit I'd most want you to take away, because once it clicks, everything after it stops being surprising.

There's no separate vision module that hands the language model a tidy list of objects. The patches are the perception. They show up as one flat run of vectors with your prompt stuck on the end, and the decoder has no tag telling it which ones came from pixels.

So if the model is going to say "number 23 took the shot", that fact has to be rebuilt by attention, over a sequence where nothing is labelled as a person.

How much is in that sequence? Eight frames at 896x504 gives a grid of [4, 36, 64], which is 9,216 patches, which collapses to 2,304 tokens.

Two things fall out of that. One token covers a 28x28 pixel block, and eight frames become four moments in time. A jump shot takes about a second, so the model gets roughly four snapshots of it.

Which parts does each stage actually train?

Stage three is instruction tuning, which teaches the model to answer a question instead of rambling on. Stage four is the domain fine-tune, which is what BARD did with 14,676 labelled basketball clips.

Two things worth pulling out of that table. First, the vision encoder is usually frozen after stage one, so the visual features a basketball model leans on were locked in before it ever saw basketball. Second, there's no column for binding, because there's no part of the model that holds "this entity, with these attributes" for a stage to train.

3. So why would binding be the thing that breaks?

Quick definition, because I'm going to keep using the word. Binding is attaching an attribute to the right entity. Not "there's a 23 and there's a white jersey in this clip", but "the 23 and the white jersey are the same guy, and he's the one who shot it".

Think of a witness who remembers that someone was wearing red and someone was called Dave, but genuinely cannot tell you whether Dave was the one in red. They saw everything. They just didn't store it as a person.

Here's why I think these models are that witness.

The representation is a grid, not a set of objects. Tokens are indexed by where they are in the frame. Nothing is indexed by who it is.

The pretraining objective never asked for it. Remember stage one. "23 in white passed to 7 in dark" and "7 in white passed to 23 in dark" are nearly the same bag of concepts. Since the negatives are unrelated images, a model can ace contrastive training while representing the play as an unordered pile of attributes. This isn't my idea, by the way. Benchmarks built to test exactly this kind of swap, like Winoground and ARO, found contrastively trained models scoring near chance.

The detail gets averaged away. One token is a 28x28 pixel block. In a wide broadcast shot, a jersey number is smaller than that, so it ends up inside a single token along with the shirt, some floor, and whatever motion blur the camera picked up.

Now compare that to a tracker. Multi-object trackers keep persistent tracks, appearance embeddings, and an explicit step that decides "this blob in frame 5 is the same person as that blob in frame 4". They have machinery for identity because identity is the entire problem they were built for. A patch grid has none of it, and no training stage ever asked it to grow any.

4. How I set up the test

BARD doesn't ask for JSON. It asks for a caption in a fixed grammar, which is what the fine-tune was trained to produce.

A few choices in the harness that matter more than they look:

• Jersey numbers are strings, not integers. "00" and "0" are different players, and parsing them as numbers quietly merges them.
• Parsing is strict. If a caption doesn't match the grammar, it gets logged as invalid with a reason. It never gets patched up into a best guess, because a patched-up output silently becomes a prediction you're scoring.
• Events are matched by Hungarian assignment, not by list position. If the model gets two events right but emits them in the other order, it shouldn't be punished for the ordering.
• The split is by game, not by clip. Clips from one game share a camera, a broadcast crew, uniforms, lighting, often the same possession. Splitting by clip leaks all of that across the boundary.

That last one seemed like ordinary hygiene when I set it up. It turned out to be the most important decision in the project, and I'll get to why.

The interventions are real pixel operations, not metadata flags. Grayscale strips the colour and keeps everything else. Scoreboard masking paints over the graphic. Temporal reordering shuffles the frames.

5. Finding 1: the number that fooled me

Here's the standard way to test for cheating. Take away a cue that shouldn't matter, and count how often the answer changes. Lots of changes means the model was leaning on it. That's the reasoning, and I believed it.

So I drained the colour out of the video, and 74.8% of the model's jersey-number answers changed. Three quarters! A model completely dependent on colour to tell players apart.

Then I checked whether any of those changed answers were actually worse. They weren't. Accuracy moved +1.9 points, which is noise.

You might be wondering how both of those can be true at once. So sort every clip by what its prediction actually did.

144 answers changed. Five went from right to wrong. Nine went from wrong to right. And 130 went from one wrong answer to a different wrong answer.

That last row is the whole problem. It's 90% of the churn, the flip rate counts every bit of it, and accuracy literally cannot see it:

flip rate    = (5 + 9 + 130) / 210 = 0.686
degradation  = (5 - 9)       / 210 = +0.019
the test     = 5 vs 9, on 14 disagreeing pairs

Here's why it had to come out this way. The model gets jersey numbers right 12.9% of the time. Always guessing the single most common number gets you 5.2%. So there's about seven points of real skill in that field, total. It could not lose 74.8% of something it never had. Perturbing the input just reshuffles a pile of wrong answers, and the flip rate dutifully reports the reshuffling as dependence.

It's like judging a student by how many answers they changed between two attempts at a test they were guessing on. Changing a guess from B to C tells you nothing about what they know.

Meanwhile action recognition, which changed on only 60 clips, less than half as many, had 28 of those go right-to-wrong against just 9 the other way. That lopsidedness is the real signal. The genuine shortcut was hiding in the quieter number the whole time.

6. Finding 2: then I fooled myself again

Having found a real effect on action, I did the obvious thing and re-ran it at four times the input resolution to pin it down. It roughly doubled: nine points of damage, p = 0.0026. Clean, quotable, done.

Except I had that validation set sitting there, built from completely different games. Genuinely independent. Running it wasn't really optional.

Minus 1.5 points. Nothing.

My first instinct was to blame sample size, and the nice thing is that's checkable rather than arguable. The test I'm using only looks at clips where the two conditions disagree. Clips that were right both times, or wrong both times, carry no information about which condition is better, the same way a taste test tells you nothing from people who liked both drinks.

First split: 28 + 9 = 37 disagreeing pairs. Validation split: 20 + 17 = 37. Identical. The test had exactly as much to work with both times. 28 against 9 is lopsided. 20 against 17 is a coin flip.

So the honest explanation is a boring one, and it's about me. I ran the high-resolution experiment because the low-resolution one already looked good. Picking your follow-up based on what already looks promising biases it upward. There's a name for it, the garden of forking paths, and knowing the name did not save me.

Pooling both independent samples gives the number I actually stand behind: 5.4 points, n = 409, p = 0.014. The effect is real. It's also about half of what I first announced, and the nine-point figure is withdrawn.

And here's the part that still bothers me. If I'd split by clip like most people do, my validation set would have shared cameras and uniforms with the test set. It would have cheerfully agreed with the inflated number, and I'd have shipped a wrong result with a small p-value attached to it.

7. Finding 3: what can it actually do?

An accuracy number is meaningless until you know what a stupid guess scores. So here's every field, against just always answering with the most common label and never looking at the video.

Action and jersey colour are comfortably ahead. Real skill. But whether an assist happened, who threw it, and whether the shot went in are all worse than not watching. You can't run a shortcut analysis on those three, because there's no ability there whose loss you'd be measuring.

Which fields fail isn't random, though.

"Was a three-pointer taken?" is one fact about the scene. Nobody has to be picked out. "Did 23 score it, assisted by 7?" needs two specific people identified and a relationship asserted between them. Every field it fails needs the second kind. Every field it passes needs only the first.

Now, the obvious objection is that jersey numbers are just too small to read at 448 pixels. Fair. So I quadrupled the resolution.

Jersey number moved 4.3 points, which doesn't clear significance. But in the exact same forward passes, jersey colour jumped 16.7 points and action jumped 7.1.

That contrast is what makes the null result worth anything. If nothing had improved, "resolution doesn't help" would be unfalsifiable, because maybe the extra pixels never delivered anything usable. Instead the detail demonstrably arrived, and the model put it to work elsewhere. Colour is a big low-frequency blob and gains the most. A number is small, high-frequency, often occluded or blurred, and on top of all that it has to be bound to the actor.

Resolution fixes legibility. It doesn't fix binding.

8. Finding 4: when breaking is good news

Last experiment. I reversed the frame order, so the model sees the same eight images backwards, and action accuracy dropped 7.1 points.

Getting that measurable took a fix first. The original implementation re-encoded a reordered video and then sampled it, which changes which frames the sampler picks. That confounds "sensitive to order" with "shown different pictures". Now it permutes the frames after sampling, so the frame set is identical and only the order changes. I verified that on all 210 clips.

Here the damage is good news, and it's worth being clear why. Colour is noise: the right answer doesn't depend on it, so leaning on it is cheating. Time isn't noise. Run a made basket backwards and the ball leaps out of the hoop, because something genuinely different happened. A model that scored the same on reversed clips would be telling you it never used time at all, just vibes from a still frame.

Which means a single "robustness" score is incoherent. It adds a number you want small to a number you want large and reports the total.

9. So what did fine-tuning actually buy?

More than the folklore predicts, honestly. The fine-tune is about twice as accurate and roughly half as jumpy under nuisance interventions. The trade everyone worries about, where you buy accuracy with shortcut reliance, just didn't show up here.

The last row is the sharper clue. Extra resolution helps the fine-tune a lot and the base model almost not at all. Same architecture, same pixels. Fine-tuning seems to have taught it where to look, not just what to say.

And then it hits the wall. It can tell you a shot was taken and what colour the shooter wore, and it essentially cannot tell you which numbered human did it.

10. What does this mean if you work in sport?

Here's the awkward part. The boundary sits in exactly the wrong place.

Think about what a play-by-play feed already gives you: event type, outcome, timestamp, and the players involved, logged by a human scorer and basically exact. A model telling you a three-pointer was made is reproducing, at 46% accuracy, something the league publishes at 100%.

The reason anyone wants video understanding was never the event type. It's everything the feed doesn't have. Positioning. Off-ball movement. Who switched, who closed out late, who was supposed to be there and wasn't. All of which need you to know who is who.

So my practical takeaway is: stop asking one model to do identity. Trackers already solve player identity reasonably well, and they solve it with architecture built for the job. Hand the identity to the model as an input instead of hoping it infers identity from pixels, and every piece is working on the part it was designed for.

That reframing also makes the fine-tuning result look better than it first did. Fine-tuning bought attention: the model learned where to look, which is exactly what the resolution row shows. What it couldn't buy is a representation that carries identity through to the output. Those are different problems, and only one of them gets fixed by collecting more basketball clips.

A guess about why more data won't fix it

Flagging this as conjecture, but it follows from the setup. The supervision is a caption. Getting "3PT Shot", "made" and "white" right earns you most of the tokens in that sentence. Jersey number is a small slice of the sequence with something like fifty plausible answers, so the cheapest way to reduce loss is to sharpen the fields that are already working. Caption-level training may simply not put much pressure on binding at all. If that's right, the fix is a loss that scores identity separately, not another 10,000 clips.

What I'd ask a vendor

If someone's selling you automated tagging from broadcast video, three questions sort a working system from a demo:

• What's the majority-class baseline for each field you report? Three of my six were below it. An accuracy number without its baseline isn't evidence of anything.
• Was the held-out set split by game or by clip? Clip splits share cameras and uniforms across the boundary. My own effect shrank by a factor of six when I moved to unseen games.
• Is player attribution scored separately from event detection? A single combined number hides the exact failure that matters, because the easy fields dominate it.

11. What could be wrong with all this

• One model family, one size. Qwen2.5-VL at 3B. Nothing here tells you how a bigger model behaves.
• My pooled colour estimate rests on two samples that disagreed. A formal test of that disagreement doesn't reach significance (Fisher p = 0.087), so I can't declare them different, but I also can't lean on the first one.
• I can't tell you whether fine-tuning created the colour shortcut or inherited it. I ran the base model to find out, and it scores 0.190 on action against a 0.152 baseline, so there's almost no headroom for the test to detect anything. My own rule from section 5 disqualifies my own comparison.
• Three of six fields sit below trivial baselines, so anything I say about those is about what the models can't do, not how they do it.
• Scoreboard masking did nothing measurable to any field, for either model. I report that as a null, not as proof it doesn't matter at other scales or with a different mask.
• Temporal reordering only tests full reversal. Partial shuffles might separate coarse from fine temporal structure. I didn't run them.

12. What I’d do with a bigger machine

Everything above is measurement. This part isn’t, and it’s also shaped by the fact that I ran all of it on one RTX 3080 with a nearly full SSD. So rather than list what someone should do, here’s what I’m actually blocked on, and what it would take to unblock it.

The one that needs more VRAM

The biggest open question is whether binding is a scale problem or a structural one. If a 30B model binds identities properly, this whole thing resolves itself with time and I can stop worrying. If it doesn’t, the interesting question becomes what a pooled patch representation can encode about entity identity, which is much less likely to fix itself.

I can’t test that here. Not slowly, at all. The 3B model at 896px peaks at 8,666 MiB of my 10,240. A 7B in bf16 needs roughly 14GB for the weights alone, before activations. A 30B needs about 60GB. So the honest situation is that the model class I’d want to check is two full steps past what fits on my desk.

What it would take: one A100 80GB, or a couple of 48GB cards, for maybe six hours. The harness already runs unmodified against any checkpoint, so it’s genuinely just the memory.

Here’s why I still lean toward structural, though. The failure is selective. The model isn’t uniformly weak. It clears its baseline comfortably on scene properties and drops below it on every single field that needs a second entity. Uniform weakness is what a scale story looks like. A clean split along the property-versus-relationship line looks more like a representational limit, and more parameters trained the same way might just sharpen the same shape.

The one that needs more disk

My n is 210 test clips, and that number isn’t a design choice. BARD has 14,676 clips. I have 2,008 of them on disk, taking 14GB, which works out to about 7MB per clip. Pulling the rest would be roughly 100GB. I have 63GB free on a drive that’s already 94% full.

So the ceiling on this study is a hard drive. That matters more than it sounds, because the weakest part of the whole piece is that my colour effect rests on two samples that disagreed with each other. With the full set I could run several genuinely independent game-level splits instead of one test and one validation, and just watch the estimate settle. That’s a much better answer than pooling two samples and hoping.

What it would take: a 2TB drive. That’s the cheapest unlock on this list by a wide margin, which is a slightly annoying thing to discover.

The one I want someone else to do

Take the decomposition from section 5 somewhere that isn’t sport. Nothing in the argument is about basketball. Any weak model paired with a quoted intervention sensitivity has the same trap sitting in it, and it would take about a day to check on a domain you already know.

If the dissociation shows up wherever a model is weak, the reporting rule is worth adopting generally. If it turns out to be a quirk of this task, I’d want to know that too, and I’d rather find out from someone who isn’t already attached to the conclusion.