When a Benchmark Stops Measuring

1The short life of a good test

A benchmark begins as a favour to everyone. Somebody assembles a few thousand problems that a competent person could solve and a machine of the day could not, publishes the set, and for a year or two the field has a shared yardstick. Papers become comparable. Claims become checkable. The number goes up and the increase means something, because nobody has yet had a reason to attack the number directly.

Then the number acquires stakes. It appears in launch posts and funding decks. Vendors quote it. Somewhere a team is assembled whose actual objective is the score rather than the capability the score was built to detect. Nothing dishonest need occur for the measurement to degrade; it is enough that effort is now aimed at the target rather than through it.

This is Goodhart's law, and it is usually invoked as a wisecrack rather than as an empirical claim about a specific measurement. Treated seriously, it makes a prediction: the correlation between benchmark score and the underlying ability should weaken over time, and it should weaken fastest for the benchmarks under the heaviest commercial pressure. That prediction is testable, and it is tested far less often than it is repeated.

2Three ways a score comes loose

The first is contamination, and it is the least interesting because it is the most obvious. Training corpora are scraped from the web; benchmarks live on the web; test items and their solutions leak into training sets, often through third-party reproductions rather than the canonical source. Decontamination pipelines exist and catch the easy cases. They catch paraphrase badly, and paraphrase is what the web does to everything.

The second is overfitting through iteration. Even with a clean split, a field that runs thousands of experiments against the same held-out set is performing gradient descent on that set by hand, one architectural decision at a time. The held-out set stops being held out the moment it starts informing choices. This is not cheating; it is the ordinary operation of a research community with a shared scoreboard, and it is why fresh test sets so often produce a sudden and unflattering drop.

The third is the one that troubles me most: the benchmark was never measuring what its name claimed. A test called reasoning may in fact be measuring familiarity with a problem genre. Perturbation studies get at this by changing surface features that should not matter, and they keep finding that performance moves when it should not. Rename the variables, change the numeric values, add an irrelevant clause, and accuracy drops in ways a person solving the problem by understanding would not exhibit.

3What the sceptics get wrong

It would be convenient to conclude that benchmarks are theatre and that reported progress is illusory. That conclusion is not supported and I do not hold it. Some of the movement is real, and the evidence for it is not the benchmark: it is that people who were not paid to like these systems started using them daily for tasks they used to do themselves. Revealed preference is a crude instrument, but it is not gameable in the same way a leaderboard is.

There is also a decent reply to the perturbation findings. Human performance degrades under surface perturbation too, sometimes substantially, and the comparison is rarely run properly. A sizeable literature on human problem solving finds that reformulating a word problem changes accuracy considerably. If we hold machines to a standard of perfect invariance that people do not meet, we are not measuring reasoning, we are measuring a philosophical preference.

So the honest position is uncomfortable and narrow. Benchmark numbers are weak evidence that degrades with age and attention. They are not zero evidence. The mistake is not using them; it is quoting a two-year-old headline figure as though it still means in 2026 what it meant when the test was fresh.

4Held-out is a property of behaviour, not of files

The most useful reframing I know is this: a test set is not held out because it sits in a different directory. It is held out because no decision has been conditioned on it. Once you have looked, once you have chosen a checkpoint or a hyperparameter or a prompt template on the basis of the score, the set has been spent. It can be re-run, but it can no longer surprise you.

Which suggests a discipline that almost nobody follows: budget your test sets like a consumable. Decide in advance how many times you will look. Keep a portion sealed for the eventual audit. Publish the number of evaluations run, not just the best one obtained. None of this is exotic; it is standard practice in clinical trials, where the cost of a spent endpoint is measured in lives rather than in press coverage.

The counterargument is that machine learning research moves too fast for that ceremony, and there is something to it. But speed is a reason to be more careful with the instruments, not less. A field that runs a thousand experiments a week and never rotates its yardstick is not moving fast; it is measuring the same illusion at higher frequency.

5Better instruments, and their costs

Several fixes are already in use and each buys something at a price. Held-out sets refreshed on a schedule, drawn from problems that postdate the training cut-off, resist contamination well but introduce distribution drift: this year's competition problems are not last year's, and part of any score change is the exam changing rather than the candidate.

Private evaluation, where the test items are never published and only aggregate scores are released, resists both contamination and iteration. It also destroys reproducibility and asks the public to trust an unaudited scorer. That is a real cost, not a rhetorical one, and it is the reason I distrust any private benchmark whose operator is also selling something.

Task-grounded evaluation is the most promising and the most expensive. Instead of asking whether a system answers questions correctly, you ask whether it completes a real job end to end: file the ticket, reconcile the ledger, fix the failing build. These evaluations are hard to construct, slow to run, and awkward to compare across organisations. They are also the only ones I have seen that predict deployment outcomes with any reliability.

6What to ask of a number

When somebody quotes a benchmark result at me now, I ask four things, and the answers are usually unavailable. When was the test set constructed relative to the training data? How many evaluations were run against it before this one was reported? Was the scoring done by the party with an interest in the outcome? And has anyone run a perturbed variant?

None of these questions are hostile. They are the ordinary questions one asks of any measurement in any empirical field, and their absence in a discipline that describes itself as empirical is the anomaly worth explaining. The absence is not conspiracy; it is incentive. Nobody is rewarded for reporting the ninety-seven evaluations that came before the good one.

I do not think benchmarks should be abandoned, and I am suspicious of people who say so, because the alternative they usually propose is vibes with better prose. What I think is that a benchmark number should be read the way one reads a company's own estimate of its own market share: informative, directional, and not the last word on anything.