Frequently Asked Questions About Kaggle DeepMind Agentic Evals at Scale Framework
23 answers covering everything from basics to advanced usage.
// Basics
What is the Democratization Problem in AI evaluation?
The Democratization Problem is the structural imbalance where approximately 30,000 AI researchers create nearly all benchmarks for a world of 30 million technical professionals and billions of end users. This means vast areas of human knowledge — industrial safety, niche legal domains, trades, specialized medicine — remain unevaluated. The consequence is cognitive jaggedness: AI performs superbly where researchers benchmark but remains mediocre in domains they don't cover.
What is the difference between a model benchmark and an agent benchmark?
A model benchmark tests a model's raw capability on a defined task (e.g., answering a trivia question). An agent benchmark tests a system that uses tools, memory, and multi-step workflows to complete complex tasks via external APIs and environments. Agent benchmarks require a harness — the execution environment and scaffolding — which can shift performance by 22%+ on identical tasks. You must explicitly state whether you are testing the model, the agent, or the harness.
What is a harness in AI evaluation?
A harness is the execution environment, scaffolding, tooling, and prompt structure surrounding a model or agent during evaluation. It includes the API version, context window size, features like context compaction, temperature settings, and tool configurations. SWE-Bench data shows the harness alone can account for 22%+ performance differences across the same frontier models on the same task, making it a critical variable to control and document.
What does hill-climbing mean in the context of AI evaluation?
Hill-climbing is the process of iteratively improving model or agent performance by measuring against a benchmark and optimizing toward higher scores. If a capability is not benchmarked, teams cannot hill-climb on it — meaning that capability will not improve systematically across model generations. This is why benchmark coverage matters: unbenchmarked domains remain cognitively jagged, with AI performance stagnating regardless of how much compute or data is applied.
What is saturation in AI benchmarks and why is it a problem?
Saturation occurs when models reach ceiling performance on a benchmark, eliminating its usefulness as a differentiating signal. Once all frontier models score 95%+ on a benchmark, it tells you nothing about which model is better. Static benchmarks inevitably saturate as models improve. PvP Game Arena architectures are designed to be permanently unsaturatable because every matchup produces a winner and a loser, maintaining signal indefinitely regardless of how capable models become.
// How To
How do I recruit domain experts to build AI benchmarks?
Use hackathons as the primary recruitment mechanism. Define focus areas and guardrails but give participants creative latitude. Provide free access to data hosting, API credits for state-of-the-art models, and writeup tools so their work is understandable and reusable. Target practitioners with deep specialized knowledge — wastewater engineers, medical specialists, tradespeople — rather than AI researchers. All outputs must be open source to maximize community value.
How do I set up a PvP Game Arena for AI model evaluation?
Design a game that isolates the specific capability you want to test (e.g., deception in negotiation, strategic planning in poker). Implement ELO scoring with Bradley-Terry pairwise scheduling to select which matchups provide the most information gain given current ranking uncertainty. Track and publish full LLM conversation logs as a dataset. Include a game visualizer or qualitative examples so results are interpretable. The structure is inherently unsaturatable because one model always wins and one always loses.
How do I write verifiable assertions for AI evaluation tasks?
Combine hard-coded checks (does the output contain a specific required element?) with LLM-as-judge scoring for qualitative dimensions like reasoning quality or safety compliance. Group assertions into Tasks and Tasks into Benchmarks. Every assertion must be independently verifiable by a third party. Expose all configuration publicly — model API version, temperature, context window — so no one can accuse you of optimizing for a specific model.
How do I deploy a Standardized Agent Exam for my team?
Provide a one-line prompt interface where developers submit their agent's system prompt and the agent takes a standardized exam returning a score on a public leaderboard. Prioritize safety-focused exam tracks first — does the agent refuse to forward sensitive data? Does it respect scope boundaries? Compare results against 500+ already-evaluated agents to contextualize performance. This covers the consumer end of the spectrum for teams that cannot build custom eval harnesses.
How do I get started with this framework if I have no evaluation experience?
Start with the Standardized Agent Exam: submit your agent's system prompt to an existing public exam endpoint and get a baseline score. Study how your agent performs relative to the 500+ already-evaluated agents on the leaderboard. Then identify one domain-specific capability gap that matters to your use case. Write 5-10 assertion-based tasks for that capability, run a pilot with 3-5 agents, and calibrate difficulty. This gives you a working eval in days, not months.
What should I publish alongside my benchmark results for full reproducibility?
Publish five artifacts: (1) the benchmark tasks and assertion definitions, (2) the complete harness configuration including model API versions, context windows, temperature, and feature flags, (3) raw LLM conversation logs, (4) the scoring methodology (ELO, Bradley-Terry, or assertion pass rates), and (5) a game visualizer or qualitative examples for interpretability. A leaderboard without these artifacts is marketing, not science. Make everything forkable and open source.
// Troubleshooting
Why do AI benchmarks go stale so quickly?
Ten or more benchmarks are published daily on arXiv, but authors typically move on to the next paper after publishing, abandoning maintenance. Meanwhile, models rapidly improve and saturate static benchmarks, eliminating their signal value. Without ongoing maintenance ownership or an inherently unsaturatable architecture (like PvP scoring), any benchmark's useful lifespan is measured in months. The framework addresses this by requiring either maintenance commitments or PvP Game Arena designs from the start.
What do I do if my benchmark is too hard and no agent can complete it?
If no agent completes the task during your pilot run with 3-5 representative agents, the benchmark produces no useful signal. Decompose the task into smaller sub-steps and test each independently. Add intermediate checkpoints that award partial credit. Reduce the scope of required tool interactions. Re-run the pilot and verify you see meaningful spread in scores across agents. Also consider whether the task requires capabilities that simply don't exist yet — in that case, bank it for future use.
What if domain experts disagree on the correct answers for a benchmark?
Even domain experts disagree, and AI cannot reliably judge innovation or creativity. Build explicit inter-expert alignment workflows: have multiple experts independently review the same tasks, measure agreement rates, and resolve disagreements through structured discussion before finalizing assertions. Document where disagreement persists and treat those as open-ended tasks scored by LLM-as-judge rather than hard-coded assertions. Never assume expert judgment is monolithic.
How do I control compute costs in PvP benchmarking?
Use Bradley-Terry pairwise scheduling to select matchups that maximize information gain given current ELO uncertainty, rather than running expensive full round-robins. Set explicit cost ceilings before starting. For games like poker that require statistical significance across hundreds of thousands of hands, budget API costs upfront. Prioritize matchups between models whose rankings are most uncertain. Publish partial results transparently if budget is exhausted before full convergence.
// Comparisons
How does the Kaggle DeepMind framework compare to running SWE-Bench?
SWE-Bench is a specific static coding benchmark; the Kaggle DeepMind framework is a meta-framework for designing evaluation systems across any domain. Crucially, the framework highlights that SWE-Bench results vary by 22%+ depending on harness configuration — proving why you must separate model, agent, and harness evaluation. The framework also addresses saturation (SWE-Bench will eventually be solved), domain coverage beyond coding, and community contribution pipelines that SWE-Bench alone does not provide.
How does the Game Arena approach compare to Chatbot Arena?
Chatbot Arena uses human preference voting to rank models on conversational quality. Game Arena uses structured game play with ELO/Bradley-Terry scoring to rank models on specific capabilities like deception, negotiation, or strategic planning. Game Arena is automated (no human judges needed per matchup), domain-specific rather than general-purpose, and produces richer behavioral data — like whether a model plays risk-averse or risk-seeking. Both are unsaturatable by design.
How does this framework compare to just using LLM-as-judge for evaluations?
LLM-as-judge is one component within the framework, not a replacement for it. The framework combines LLM-as-judge with hard-coded assertion checks, PvP Game Arena scoring, standardized harness documentation, and domain expert recruitment. Using LLM-as-judge alone risks unreliable scoring on creative or innovative outputs, lacks verifiability guarantees, and doesn't address saturation, the Democratization Problem, or the harness confound that this framework systematically resolves.
// Advanced
Can I use this framework to evaluate proprietary enterprise AI systems?
Yes. Define your evaluation target explicitly (model, agent, or harness), identify domain expertise gaps specific to your enterprise (e.g., internal compliance protocols), and choose between assertion-based benchmarks for proprietary knowledge and PvP arenas for competitive capability testing. You can adjust transparency requirements — the framework supports fully open source, reproducible but proprietary, or hybrid configurations. Internal hackathons can recruit domain experts from within your organization.
How do I measure whether my eval system is actually working?
Track three signals: (1) differentiation — do different agents produce meaningfully different scores, or does everything cluster? (2) stability — do rankings remain consistent across repeated runs with the same configurations? (3) predictive validity — do benchmark scores correlate with real-world agent performance on the tasks you care about? If differentiation collapses, recalibrate on the Difficulty Spectrum. If stability is low, check for harness variability. If predictive validity is weak, your assertions may not capture the right capabilities.
How should I structure assertions for safety-critical AI evaluations?
For safety-critical domains, prioritize hard-coded assertions over LLM-as-judge scoring — safety compliance must be binary and verifiable, not subjective. Test both positive cases (does the model recommend the correct emergency shutdown procedure?) and adversarial edge cases (does the model maintain safe recommendations when presented with misleading context?). Recruit domain experts with real incident experience to author edge-case scenarios. Publish assertion logic openly so the safety community can audit and extend coverage.
Can this framework handle multi-modal AI evaluations?
The framework's architecture — separating model, agent, and harness; using assertion-based tasks with LLM-as-judge; applying PvP arenas — is modality-agnostic in principle. For multi-modal evals, your harness must handle image, audio, or video inputs consistently across all evaluated models. Your assertions need modality-appropriate checks (e.g., does the model correctly identify a safety hazard in an image?). The standardized harness documentation becomes even more critical because multi-modal API configurations vary significantly across providers.
How many benchmarks should I create for a comprehensive eval program?
There is no fixed number. Focus on coverage of the capabilities that matter for your use case rather than a benchmark count target. Map out all critical capabilities, identify which are already covered by existing public benchmarks, and build new benchmarks only for gaps — especially proprietary novel domains. For each benchmark, calibrate on the Difficulty Spectrum. A small number of well-calibrated, maintained benchmarks provides more signal than dozens of stale ones.