Wrong Answer, Different Wrong Answer

People are starting to use vision-language models for action tracking: give one a video clip and ask what happened and who did it. I built a test to check whether one of these models was really watching. The test came back positive. When I broke the number into its parts, it turned out to be measuring almost nothing.

1. What I was testing

Point a vision-language model at six seconds of a pick-and-roll and ask what happened. You get back something like this:

“A player with jersey number 23 and jersey_color white made a 3PT Shot which result was made and was assisted by other player with jersey number 7.”

That's a structured event pulled straight from raw footage, with no custom tracking pipeline behind it. It reads like a scout wrote it. If it's reliable, you could tag a whole season of film with it.

The trouble is that broadcast footage is full of things that correlate with the right answer without being the right answer. The home team wears white, so team identity is mostly a colour. The scoreboard sits in the same corner of every frame. Highlight clips are cut around exactly one scoring event, so "something scored" is a safe bet before you look at anything.

A model can do well on all of that without ever tracking a person, and tracking the person is the reason you wanted the model.

So I built a harness. Remove a cue that shouldn't matter, see what the model does, and use that to separate real understanding from correlation. It's a standard approach and I had no reason to doubt it.

It turns out the standard approach can rank the cheating backwards. That was the more interesting result, so it's most of what follows.

2. How the model is built

The failure makes more sense once you know how the model was put together, because no stage of training ever asked it to do the thing it can't do. There are four stages.

Contrastive pretraining

The first stage is contrastive pretraining, the recipe CLIP made popular. You take a big batch of images and their captions, and you train two encoders so that each image lands near its own caption in a shared space. The training signal is a grid of similarity scores.

That gives you an encoder with a good sense of what's in a scene. But look at what it never has to do. The things it pushes away are other images in the batch: a beach, a dog, a kitchen. It learns to tell a basketball court from a kitchen. There's almost no pressure to tell one arrangement of players from another.

The bridge to the language model

There were a few designs for this, including gated cross-attention and learned query bottlenecks. The one that won is the simplest: take the patch embeddings from the vision encoder, run them through a small network that maps them into the language model's word-embedding space, and let the decoder treat them as words.

In Qwen2.5-VL the merge and the projection are one operation. Four patches from a 2x2 block get glued into a 5,120-long vector, then a small MLP squeezes that into one 2,048-dim token. The vision encoder works in 1,280 dims across 32 layers; the decoder works in 2,048.

That small MLP is the whole interface between seeing and speaking, which is thinner than I expected.

What the decoder sees

This is the part that explained everything after it for me.

There's no separate vision module handing the language model a list of objects. The patches are the perception. They arrive as one flat run of vectors with your prompt on the end, and nothing tells the decoder which ones came from pixels.

So if the model is going to say "number 23 took the shot", attention has to rebuild that fact from a sequence where nothing is labelled as a person.

Eight frames at 896x504 gives a grid of [4, 36, 64], which is 9,216 patches, which collapses to 2,304 tokens.

Two things fall out of that. One token covers a 28x28 pixel block, and eight frames become four moments in time. A jump shot takes about a second, so the model gets roughly four snapshots of it.

What each stage trains

Stage three is instruction tuning, which teaches the model to answer the question instead of rambling. Stage four is the domain fine-tune, which is what BARD did with 14,676 labelled basketball clips.

Two things stand out in that table. The vision encoder is usually frozen after stage one, so the visual features a basketball model relies on were fixed before it ever saw basketball. And there's no column for binding, because no part of the model holds "this entity, with these attributes" for a stage to train.

3. Why binding breaks

A definition first, since I'll keep using the word. Binding is attaching an attribute to the right entity. Knowing there's a 23 and a white jersey in the clip is not enough. Binding means knowing the 23 and the white jersey are the same player, and that he took the shot.

A model without binding can know every attribute in the scene and still not know which attributes go together.

I think these models fall into that gap for three reasons.

The representation is a grid, not a set of objects. Tokens are indexed by where they are in the frame. Nothing is indexed by who it is.

The pretraining objective never asked for it. This is the contrastive pressure from stage one. "23 in white passed to 7 in dark" and "7 in white passed to 23 in dark" contain nearly the same concepts. Since the negatives are unrelated images, a model can do well in contrastive training while representing the play as an unordered pile of attributes. This isn't a new observation. Benchmarks built to test exactly this kind of swap, like Winoground and ARO, found contrastively trained models scoring near chance.

The detail gets averaged away. One token is a 28x28 pixel block. In a wide broadcast shot, a jersey number is smaller than that, so it ends up inside a single token along with the shirt, some floor, and whatever motion blur the camera picked up.

Compare that with a tracker. Multi-object trackers keep persistent tracks, appearance embeddings, and an explicit step that decides whether the player in frame 5 is the same one from frame 4. They have machinery for identity because identity is the problem they were built for. A patch grid has none of that, and no training stage asked it to develop any.

4. How I set up the test

BARD doesn't ask for JSON. It asks for a caption in a fixed grammar, which is what the fine-tune was trained to produce.

A few harness choices matter more than they look:

• Jersey numbers are strings, not integers. "00" and "0" are different players, and parsing them as numbers quietly merges them.
• Parsing is strict. If a caption doesn't match the grammar, it gets logged as invalid with a reason. It never gets patched up into a best guess, because a patched-up output silently becomes a prediction you're scoring.
• Events are matched by Hungarian assignment, not by list position. If the model gets two events right but emits them in the other order, it shouldn't be punished for the ordering.
• The split is by game, not by clip. Clips from one game share a camera, a broadcast crew, uniforms, lighting, often the same possession. Splitting by clip leaks all of that across the boundary.

The last one felt like routine hygiene when I set it up. It turned out to be the most important decision in the project.

The interventions are real pixel operations, not metadata flags. Grayscale strips the colour and keeps everything else. Scoreboard masking paints over the graphic. Temporal reordering shuffles the frames.

5. Finding 1: the number that fooled me

The standard test for cheating is to remove a cue that shouldn't matter and count how often the answer changes. If it changes a lot, the model was leaning on that cue. I believed that.

So I removed the colour from the video, and 74.8% of the model's jersey-number answers changed. That looked like a model that depended on colour to tell players apart.

Then I checked whether the changed answers were worse. They weren't. Accuracy moved +1.9 points, which is noise.

Both are true at once. Sorting every clip by what its prediction did shows how.

144 answers changed. Five went from right to wrong. Nine went from wrong to right. And 130 went from one wrong answer to a different wrong answer.

That last group is the problem. It's 90% of the churn. The flip rate counts all of it, and accuracy can't see any of it:

flip rate    = (5 + 9 + 130) / 210 = 0.686
degradation  = (5 - 9)       / 210 = +0.019
the test     = 5 vs 9, on 14 disagreeing pairs

It had to come out this way. The model gets jersey numbers right 12.9% of the time. Always guessing the most common number gets 5.2%. So there are about seven points of real skill in that field. It could not lose 74.8% of something it never had. Changing the input just shuffles a pile of wrong answers, and the flip rate reports the shuffling as dependence.

A changed guess tells you nothing about what the model knows.

Action recognition changed on only 60 clips, less than half as many, but 28 of those went right-to-wrong against 9 the other way. That imbalance is the real signal. The genuine shortcut was in the smaller number.

6. Finding 2: then I fooled myself again

With a real effect on action, I did the obvious thing and re-ran it at four times the input resolution to pin it down. The effect roughly doubled: nine points of damage, p = 0.0026. Clean and quotable.

But I also had a validation set built from completely different games, which made it properly independent. I had to run it.

Minus 1.5 points. Nothing.

My first instinct was to blame sample size, and that can be checked. The test only looks at clips where the two conditions disagree. Clips that were right both times, or wrong both times, say nothing about which condition is better.

First split: 28 + 9 = 37 disagreeing pairs. Validation split: 20 + 17 = 37. Identical. The test had exactly as much to work with both times. 28 against 9 is lopsided. 20 against 17 is a coin flip.

So the explanation is a boring one, and it's about me. I ran the high-resolution experiment because the low-resolution one already looked good. Choosing your follow-up based on what looks promising biases it upward. It has a name, the garden of forking paths, and knowing the name didn't stop me.

Pooling both independent samples gives the number I actually stand behind: 5.4 points, n = 409, p = 0.014. The effect is real. It's also about half what I first reported, and I've withdrawn the nine-point figure.

This is the part that still bothers me. If I'd split by clip, as most people do, the validation set would have shared cameras and uniforms with the test set. It would have agreed with the inflated number, and I'd have published a wrong result with a small p-value.

7. Finding 3: what it can actually do

An accuracy number means little until you know what a dumb guess scores. Here's every field against always answering with the most common label without looking at the video.

Action and jersey colour are comfortably ahead, so there's real skill there. But whether an assist happened, who made it, and whether the shot went in all score worse than not watching. You can't run a shortcut analysis on those three, because there's no ability there to lose.

Which fields fail isn't random.

"Was a three-pointer taken?" is one fact about the scene. Nobody has to be picked out. "Did 23 score it, assisted by 7?" requires identifying two specific people and a relationship between them. Every field the model fails needs the second kind of answer. Every field it passes needs only the first.

The obvious objection is that jersey numbers are too small to read at 448 pixels. Fair, so I quadrupled the resolution.

Jersey number moved 4.3 points, which isn't significant. In the same forward passes, jersey colour jumped 16.7 points and action jumped 7.1.

That contrast is what makes the null result useful. If nothing had improved, "resolution doesn't help" couldn't be tested, because maybe the extra pixels never carried anything usable. Instead the detail clearly arrived and the model used it elsewhere. Colour is a large, low-frequency area and gains the most. A number is small, high-frequency, often occluded or blurred, and it also has to be bound to the right player.

Resolution fixes legibility. It doesn't fix binding.

8. Finding 4: when breaking is good news

Last experiment. I reversed the frame order, so the model sees the same eight images backwards, and action accuracy dropped 7.1 points.

Measuring that properly needed a fix first. The original code re-encoded the reordered video and then sampled it, which changes which frames get picked. That mixes up "sensitive to order" with "shown different pictures". The fixed version permutes the frames after sampling, so the frame set is identical and only the order changes. I checked that on all 210 clips.

Here the drop is good news. Colour is a nuisance variable: the right answer doesn't depend on it, so leaning on it is cheating. Time is different. Play a made basket backwards and the ball jumps out of the hoop, because a different event happened. A model that scored the same on reversed clips would be telling you it never used time at all and was guessing from still frames.

So a single "robustness" score doesn't make sense. It adds a number you want small to a number you want large and reports the total.

9. What fine-tuning bought

Fine-tuning helped more than I expected. The fine-tune is about twice as accurate and roughly half as sensitive to nuisance interventions. The trade people worry about, buying accuracy with shortcut reliance, didn't show up here.

The last row is the more telling one. Extra resolution helps the fine-tune a lot and the base model barely at all, with the same architecture and the same pixels. Fine-tuning seems to have taught it where to look, as well as what to say.

Then it stops. It can tell you a shot was taken and what colour the shooter wore, and it essentially can't tell you which numbered player took it.

10. What this means if you work in sport

The awkward part is where the boundary falls.

A play-by-play feed already gives you event type, outcome, timestamp, and the players involved, logged by a human scorer and basically exact. A model telling you a three-pointer was made is reproducing, at 46% accuracy, something the league publishes at 100%.

People want video understanding for what the feed doesn't have: positioning, off-ball movement, who switched, who closed out late, who was supposed to be somewhere and wasn't. All of that requires knowing who is who.

So my practical advice is to stop asking one model to do identity. Trackers already handle player identity reasonably well, with architecture built for it. Give the model identity as an input instead of hoping it infers it from pixels, and each piece works on the part it was designed for.

Seen that way, the fine-tuning result looks better than it first did. Fine-tuning taught the model where to look, which is what the resolution row shows. It couldn't give it a representation that carries identity through to the output. Those are separate problems, and only one of them gets fixed by collecting more basketball clips.

A guess about why more data won't fix it

This is conjecture, but it follows from the setup. The supervision is a caption. Getting "3PT Shot", "made" and "white" right earns most of the tokens in that sentence. Jersey number is a small part of the sequence with something like fifty plausible answers, so the cheapest way to reduce loss is to sharpen the fields that already work. Caption-level training may put very little pressure on binding. If so, the fix is a loss that scores identity separately, not another 10,000 clips.

What I'd ask a vendor

If someone is selling you automated tagging from broadcast video, three questions separate a working system from a demo:

• What's the majority-class baseline for each field you report? Three of my six were below it. An accuracy number without its baseline isn't evidence of anything.
• Was the held-out set split by game or by clip? Clip splits share cameras and uniforms across the boundary. My own effect shrank by a factor of six when I moved to unseen games.
• Is player attribution scored separately from event detection? A single combined number hides the exact failure that matters, because the easy fields dominate it.

11. What could be wrong with all this

• One model family, one size. Qwen2.5-VL at 3B. Nothing here tells you how a bigger model behaves.
• My pooled colour estimate rests on two samples that disagreed. A formal test of that disagreement doesn't reach significance (Fisher p = 0.087), so I can't declare them different, but I also can't lean on the first one.
• I can't tell you whether fine-tuning created the colour shortcut or inherited it. I ran the base model to find out, and it scores 0.190 on action against a 0.152 baseline, so there's almost no headroom for the test to detect anything. My own rule from section 5 disqualifies my own comparison.
• Three of six fields sit below trivial baselines, so anything I say about those is about what the models can't do, not how they do it.
• Scoreboard masking did nothing measurable to any field, for either model. I report that as a null, not as proof it doesn't matter at other scales or with a different mask.
• Temporal reordering only tests full reversal. Partial shuffles might separate coarse from fine temporal structure. I didn't run them.

12. What I’d do with a bigger machine

Everything above is measurement. This part isn't, and it's shaped by the fact that I ran all of it on one RTX 3080 with a nearly full SSD. So here's what I'm actually blocked on, and what it would take to get unblocked.

The one that needs more VRAM

The biggest open question is whether binding is a scale problem or a structural one. If a 30B model binds identities properly, this resolves itself with time. If it doesn't, the question becomes what a pooled patch representation can encode about entity identity, which is much less likely to fix itself.

I can't test that here at all, slowly or otherwise. The 3B model at 896px peaks at 8,666 MiB of my 10,240. A 7B in bf16 needs roughly 14GB for the weights alone, before activations. A 30B needs about 60GB. The model class I'd want to check is two full steps past what fits on my desk.

What it would take: one A100 80GB, or a couple of 48GB cards, for maybe six hours. The harness runs unmodified against any checkpoint, so it really is just memory.

I still lean toward structural, because the failure is selective. The model isn't uniformly weak. It clears its baseline comfortably on scene properties and drops below it on every field that needs a second entity. Uniform weakness would suggest scale. A clean split along the property-versus-relationship line looks more like a representational limit, and more parameters trained the same way might just sharpen the same pattern.

The one that needs more disk

My n is 210 test clips, and that number isn’t a design choice. BARD has 14,676 clips. I have 2,008 of them on disk, taking 14GB, which works out to about 7MB per clip. Pulling the rest would be roughly 100GB. I have 63GB free on a drive that’s already 94% full.

So the ceiling on this study is a hard drive. That matters, because the weakest part of the piece is that my colour effect rests on two samples that disagreed. With the full set I could run several independent game-level splits instead of one test and one validation, and watch the estimate settle. That's much better than pooling two samples and hoping.

What it would take: a 2TB drive. It's by far the cheapest item on this list, which is a little annoying to realize.

The one I want someone else to do

Take the decomposition from section 5 to a field that isn't sport. Nothing in the argument is specific to basketball. Any weak model reported with an intervention sensitivity has the same trap in it, and checking would take about a day in a domain you already know.

If the pattern shows up wherever a model is weak, the reporting rule is worth adopting generally. If it's specific to this task, I'd want to know that too, and I'd rather hear it from someone who isn't attached to the conclusion.