Benchmark
eval · evaluation set · leaderboard
Kurz gesagt
A benchmark is a fixed set of tasks with known answers, used to score and compare models. Public benchmarks give a rough capability ranking but are heavily optimised for, vulnerable to training-data contamination, and shaped like short well-specified problems rather than real production work.
Benchmarks exist because model comparison needs a common yardstick, and they are genuinely useful for that: they establish a rough ordering and catch large regressions.
They also fail in predictable ways, all traceable to a single dynamic — once a number becomes a target, it stops being a measurement.
- Contamination. Public benchmarks are on the public web, and training corpora are scraped from the public web.
- Optimisation pressure. Vendors tune against the benchmarks they will be judged on. This is rational and it inflates scores relative to untested capability.
- Task shape. Short, single-turn, unambiguous, with one correct answer. Little production work looks like that.
- Selective reporting. Every launch shows the benchmarks it wins.
- Saturation. Once every credible model scores above 90%, the remaining differences are noise.
The alternative is not to abandon benchmarks but to add your own. A few hundred real requests from your traffic, with correct answers agreed by people who know the domain, scored the same way every time. That set answers the question you actually have — which model is better for this — and it is the only one that survives a model upgrade.
Teams that build one routinely find the ordering differs from the leaderboard, sometimes in favour of the cheaper model.
Häufige Fragen
- Why do benchmark leaders sometimes disappoint in production?
- Benchmarks reward short, unambiguous, single-turn tasks. Production work is long, ambiguous, multi-turn and adversarial. The correlation between the two is real but loose.
- What is benchmark contamination?
- Benchmark questions and answers appearing in a model's training data, usually because they were published on the web. A contaminated benchmark measures recall rather than capability, and it cannot be fully ruled out.