How to Build Evals That Never Saturate
For AI research and platform teams · Based on Kaggle DeepMind Agentic Evals at Scale Framework
// TL;DR
If your team keeps shipping benchmarks that saturate within a year — or gets accused of tuning configs to favor your own model — this framework fixes both. Use PvP Game Arena architectures with ELO/Bradley-Terry scoring for permanently unsaturatable evals, isolate whether you're testing the model, agent, or harness (harness alone drives 22%+ swings), and control compute costs with pairwise scheduling instead of full round-robins. Publish full reproducibility artifacts so results are science, not marketing. Use it whenever you build long-running evaluation programs or benchmark frontier models credibly.
Why do your benchmarks keep going stale?
Because static leaderboards inevitably saturate — models reach ceiling performance and the signal disappears. Over ten benchmarks are published daily on arXiv, and authors move on to the next paper, leaving leaderboards irrelevant. If you want an evergreen eval, you need an architecture that can't be maxed out. That's the PvP (model vs. model) Game Arena: using ELO or Bradley-Terry scoring, there is always a winner and a loser, so no model reaches a permanent ceiling. Design for unsaturatability from the start, or assign explicit ongoing maintenance ownership — those are your only two credible options.
How do you decide between PvP and assertion-based architectures?
Base it on saturation risk. If the task space is finite and models will likely solve it within 12-18 months, use a PvP Game Arena with ELO scoring. If the task space is open-ended or the value comes from proprietary domain knowledge, use an assertion-based benchmark with LLM-as-judge scoring. Never ship a static leaderboard as your only output. A well-designed game also isolates a single capability — for multi-step deception, a negotiation game with hidden information — and surfaces emergent model personalities like risk-seeking versus risk-averse behavior as secondary signal.
How do you avoid accidentally testing the harness?
Explicitly separate the model, the agent, and the harness, and lock two as controlled constants. Per SWE-Bench data, harness configuration alone accounts for 22%+ performance differences across the same frontier models on identical tasks. State upfront which of the three you're evaluating. Then build or select a standardized harness and document it completely: model API version, context window, temperature, and whether features like context compaction are enabled. Use a consistent LLM model proxy layer so every model is called identically — this is also what protects you from accusations that you tuned configs to favor your own model.
How do you keep PvP compute costs under control?
Don't run full round-robins at statistical significance — one poker evaluation required 400,000 hands. Instead, use Bradley-Terry pairwise scheduling to select which matchups to run next, prioritizing those with the most information gain given current ELO uncertainty. Set explicit cost ceilings before you start, and for agentic tasks, factor token cost into task length so long tasks don't become prohibitively expensive at scale. Always track and publish the full LLM conversation logs as a dataset for community analysis.
How do you make results credible instead of marketing?
Publish full reproducibility artifacts: the benchmark tasks, the harness config, the raw conversation logs, the ELO/score methodology, and a game visualizer or qualitative examples. Make everything forkable and open source. A leaderboard without these artifacts is marketing, not science — and third parties should be able to reproduce every number you report. To scale coverage beyond your own team, run hackathons with clear guardrails but creative latitude, providing free data hosting, API credits, and writeup tools, with all outputs open source. That's how you turn the Democratization Problem into a strength.
Next step: Audit your current benchmark portfolio. Flag every static leaderboard at saturation risk within 18 months, re-architect the finite ones into PvP Game Arenas, and add Bradley-Terry scheduling plus full artifact publication to your next release.
// FREQUENTLY ASKED QUESTIONS
How does Bradley-Terry pairing reduce evaluation cost?
Bradley-Terry pairing selects the next matchup with the most information gain given current ELO uncertainty, so you skip low-value games instead of running full round-robins. Since round-robins at statistical significance can require hundreds of thousands of instances — one poker eval needed 400,000 hands — pairwise scheduling dramatically cuts the number of games and thus API cost while preserving ranking confidence.
Can any benchmark be made unsaturatable?
No. Only tasks that can be reframed as competitive (model vs. model) can be wrapped in a PvP structure with ELO scoring for permanent unsaturatability. Inherently single-output correctness checks can't become PvP — for those, assign ongoing maintenance ownership and periodically raise difficulty to keep signal alive. Choose architecture based on saturation risk before you build.
How do I prove my benchmark isn't biased toward my own model?
Run all models under identical harness conditions via a consistent LLM model proxy layer, and publish full configuration — API versions, context windows, temperature, and whether features like context compaction are enabled. Then release raw conversation logs, tasks, and scoring methodology as forkable open source. Reproducibility by any third party is the only credible defense against bias accusations.