Analysis: public benchmarks stopped predicting production quality — how to build an eval set that does
Leaderboard scores and shipped reliability have been drifting apart for two years. A working method for building the small, boring, private eval set that actually tells you whether to ship.
Respuesta rápida
Why do models with high benchmark scores still fail in production, and how should teams evaluate models themselves?
Public benchmarks measure narrow, static, widely-published tasks, and they are increasingly contaminated by training data and optimised for directly. Production quality depends on your prompts, your documents, your tool schemas and your failure tolerance, none of which any leaderboard measures. The practical answer is a private eval set of 50 to 300 real cases with recorded expected behaviour, run on every model and prompt change, scored by exact checks where possible and by a rubric-driven model judge where not.
Claves
- A leaderboard answers 'which model is generally stronger'. A deployment decision needs 'which model is better at my task, on my data, under my constraints'. They are different questions and only one of them is answerable by someone else.
- Fifty well-chosen cases from your own traffic outperform any public benchmark for predicting whether a change is safe to ship.
- Split the eval by failure consequence, not by topic. Cases where a wrong answer is expensive deserve their own gate.
- Deterministic checks first, model judges second. Anything you can assert with code should never be delegated to a judge.
- The eval set is a living asset: every production incident should end with a new case added to it.
There is a specific disappointment that most teams shipping with language models have now experienced. A new model arrives with better numbers on every chart. It gets swapped in. Within a week, support tickets are up, an internal user reports that the summaries got worse, and someone quietly reverts.
Nothing was faked. The benchmark scores were real, and the model probably is stronger in general. The mistake was in the inference: a leaderboard answers a question that no deployment ever asks.
What a benchmark is actually measuring
Public benchmarks share a set of properties that are helpful for science and unhelpful for decisions.
They are static. A benchmark that changes cannot be compared over time, so the items stay fixed. That means the same questions are answered in thousands of papers, blog posts and repositories, which puts them on the open web, which puts them in training corpora. This is contamination, and it is not an accusation of misconduct — it is the default outcome of publishing a test and then training on the internet. The effect is that scores drift upward faster than capability does.
They are narrow. Multiple-choice knowledge, competition mathematics, isolated coding problems and short-form reasoning are all legitimate probes, but the correlation between them and "handles our customers' badly-punctuated support emails without inventing a refund policy" is weak, and nobody has ever claimed otherwise.
They are optimised for. Once a number becomes the industry's shorthand for progress, it enters every training and evaluation loop as a target. Goodhart's law does the rest. This does not require anyone to cheat; it only requires everyone to care.
They discard the constraints that decide production quality: your latency budget, your context length, your tool schemas, your retrieval quality, your prompt, your output format, and how bad a specific kind of error is for you.
Human-preference leaderboards fix some of this and introduce their own distortions. They aggregate over a population whose tasks are not yours, and they reward the qualities that read as good in a side-by-side glance: fluent structure, confident framing, generous length. If your product needs terse, literal, refuse-when-unsure behaviour, a preference ranking can point you in exactly the wrong direction.
The alternative is smaller than people expect
The reason teams do not build private evals is that they imagine something enormous. It is not. A useful eval set is typically 50 to 300 cases, and the first version can be assembled in a day.
Step one: take cases from real traffic. Not invented examples. Pull from logs — actual user messages, actual documents, actual tool call sequences. Invented cases share the blind spots of the person who invented them.
Step two: stratify by consequence, not by topic. The instinct is to group by subject matter. The useful axis is what a wrong answer costs:
- Silent-wrong — plausible, confident, incorrect. The most expensive class, and the one worth over-sampling.
- Refusal-should-have-happened — the model answered when the correct behaviour was to decline or escalate.
- Format breakage — valid content in a shape that breaks the parser downstream.
- Cosmetic — tone, length, ordering. Real, cheap.
A model that improves on cosmetic cases and regresses on silent-wrong cases has a better average score and is a worse product. Only a stratified set surfaces that.
Step three: record expected behaviour, not an expected string. For most cases the correct label is a set of assertions — must cite document 3, must not mention pricing, must return valid JSON with these fields, must decline. Assertions survive rewording; golden strings do not.
Step four: run deterministic checks first. Schema validity, required fields, forbidden strings, numeric tolerances, citation presence, latency, token cost. This tier is cheap, exact, and catches a surprising share of real regressions. Anything a assert can decide should never reach a judge.
Step five: use a model judge only for what is left. Faithfulness to a source, tone, completeness, whether an explanation is actually responsive. The judge needs a written rubric with concrete criteria, a reference answer where one exists, and randomised ordering when comparing two outputs. Report agreement between the judge and a human on a sample; if it is poor, the rubric is the problem, not the judge.
Reading the results without fooling yourself
Two failure modes account for most bad decisions made from eval data.
The first is treating a small difference as a difference. With 100 cases, a change from 82% to 85% is well inside noise. Either report a confidence interval, or run the set several times and look at the variance, or resist the urge to declare a winner. Sampling temperature alone can move a small set by several points.
The second is averaging away the thing you care about. An aggregate pass rate hides the shape of the failures. The useful output of an eval run is not a number but a table: pass rate per stratum, plus the diff of which specific cases flipped. Cases that flipped from pass to fail are the entire signal — read them.
Where the eval set comes from over time
The first version comes from logs. The durable version comes from incidents. Every production failure that reaches a human should end with a case added to the set, with the expected behaviour recorded. This turns support load into regression coverage, and it is the mechanism that makes the eval set specific to your product rather than a generic quality proxy.
The same discipline applies to retrieval-based systems, where an extra layer is required: evaluate retrieval separately from generation. If the answer is wrong because the right document was never retrieved, no amount of model swapping fixes it, and an end-to-end score will keep telling you the model is the problem.
What benchmarks are still good for
None of this makes public evaluation worthless, and the reflex to dismiss leaderboards entirely is its own error.
They are a reasonable screen: if you need to pick three candidates from twenty models to test properly, published scores plus context length plus price will pick a defensible three. They are a useful field-level trend line — the movement of open-weight models relative to frontier models over the last two years is a real signal and would be hard to see any other way. And a small number of them, particularly agentic and repository-level benchmarks that resist memorisation by requiring an executed result, are genuinely informative about capability shifts.
The rule is narrow and worth stating plainly: use public benchmarks to decide what to test, and your own eval set to decide what to ship. Teams that reverse those two spend a lot of time explaining to their users why the upgrade made things worse.
Preguntas frecuentes
- Are public benchmarks useless?
- No. They are useful for coarse screening — deciding which three models are worth testing at all — and for tracking the field over time. They stop being useful the moment they are used as a substitute for testing on your own task.
- What is benchmark contamination?
- Test items, or near-duplicates of them, appearing in a model's training data. Because most public benchmarks are on the open web, this happens by default, and it inflates scores in a way that does not transfer to unseen work.
- How many eval cases are enough?
- Fifty is enough to catch large regressions and is achievable in an afternoon. A few hundred, stratified by scenario, is enough to detect the smaller quality shifts that matter between similar models. Beyond about a thousand, the cost of maintaining labels usually outgrows the added signal.
- Is using a model to grade another model's output reliable?
- It is reliable enough for relative comparisons when the rubric is specific, the judge sees a reference answer, and position and verbosity biases are controlled. It is not reliable as an absolute quality score, and it should never grade something a unit test could check.
- How often should the eval run?
- On every prompt change, every model version change, every tool schema change, and on a schedule against production to catch drift in the data rather than the model.