Kaggle DeepMind Agentic Evals at Scale Framework

// When should you use the Agentic Evals at Scale Framework?

Use this skill when you need to build, audit, or expand an AI evaluation or benchmarking program — especially when evaluations are stale, opaque, narrowly authored, or fail to cover domain-specific knowledge that mainstream AI labs ignore. Also applicable when deciding how to structure agent testing before production deployment.

// What do you need before building an agentic evaluation?

• evaluation_targetrequired
  What is being evaluated — a model, an agent, a harness, or a combination? Be explicit.
• domain_or_capabilityrequired
  The specific capability or domain the benchmark should cover (e.g., wastewater treatment safety protocols, poker deception, coding under ambiguity).
• audience_typerequired
  Who will build and consume these evals — research labs, enterprise teams, consumer agent builders, or community contributors?
• saturation_risk
  Is there a risk the benchmark will be solved/saturated quickly? Describe the current state of model performance on similar tasks.
• transparency_requirements
  What level of openness is required — fully open source, reproducible configs, public leaderboard, or proprietary?

// What core principles drive scalable agentic evaluation?

// How do you design and deploy an agentic eval step by step?

1. 1

   Define what is actually under test

   Explicitly separate: (a) the model, (b) the agent, (c) the harness. Do not conflate them. A benchmark that mixes these produces misleading results. State upfront which of the three you are evaluating and lock the other two as controlled constants.

2. 2

   Identify the domain expertise gap

   Ask: is this capability covered by existing benchmarks? If the knowledge lives only in a practitioner's head or proprietary field (like wastewater treatment, niche legal domains, rare engineering disciplines), treat it as a Proprietary Novel Data Set opportunity. Recruit that domain expert as the benchmark author, not an AI researcher.

3. 3

   Choose the eval architecture based on saturation risk

   If the task space is finite and models will likely saturate it within 12-18 months, use a PvP / Game Arena architecture with ELO scoring. If the task space is open-ended or domain knowledge is proprietary, use an assertion-based benchmark with LLM-as-judge scoring. Never use a static leaderboard as your only output.

4. 4

   Design assertions and tasks with explicit verifiability

   Write assertions that any third party can verify independently. Combine hard-coded checks (does the output contain X?) with LLM judging for qualitative dimensions. Group assertions into Tasks; group Tasks into Benchmarks. Expose all configuration publicly so no one can accuse you of optimizing for a specific model.

5. 5

   Calibrate difficulty on the Difficulty Spectrum

   Run a small pilot with 3-5 representative agents. If none complete the task, it is too hard. If all score above 90%, it is too easy. Adjust task complexity until the pilot shows meaningful spread across agents. For agentic tasks, factor in token cost — tasks that are too long become prohibitively expensive at scale.

6. 6

   Build or select a standardized harness and document it completely

   Use a consistent LLM model proxy layer so all models are called identically. Document: model API version, context window used, whether features like context compaction are enabled or disabled, temperature, and any other configs. The wastewater engineer's benchmark and a frontier lab's benchmark must run under identical harness conditions to be comparable.

7. 7

   Deploy a Standardized Agent Exam for consumer/low-resource users

   For agents that need a quick baseline before deployment (e.g., an OpenAI operator agent or a custom automation), provide a one-line prompt interface that returns a score on a public leaderboard. This covers the consumer end of the spectrum — most agent builders are not running any evals before deploying. Prioritize safety-focused exam tracks.

8. 8

   Use hackathons to channel domain expertise at scale

   Hackathons are the mechanism to convert the Democratization Problem into a strength. Define guardrails (focus areas, e.g., five specific cognitive faculties) but give participants creative latitude. Provide free access to: data hosting, API credits for state-of-the-art models, and writeup tools so work is understandable and reusable by others. All outputs must be open source.

9. 9

   Apply Bradley-Terry pairwise scheduling to control compute costs

   Running full round-robins at statistical significance is extremely expensive (e.g., 400,000 poker hands for one game). Use Bradley-Terry pairwise comparison to select which matchups to run next — prioritize matchups with the most information gain given current ELO uncertainty. Always track and publish the full LLM conversation logs as a dataset for community analysis.

10. 10

    Publish results with full reproducibility artifacts

    Publish: the benchmark tasks, the harness config, the raw LLM conversation logs, the ELO/score methodology, and a game visualizer or qualitative examples where possible. A leaderboard without these artifacts is marketing, not science. Make everything forkable and open source.

// What does the Agentic Evals framework look like in practice?

// What mistakes should you avoid when building agentic evals?

• Benchmarking the model when you are actually benchmarking the harness — the same six frontier models can differ by 22%+ on identical tasks depending on harness configuration.
• Optimizing benchmark configuration for your own model (e.g., enabling proprietary API features only for your model) and publishing results as neutral. Always use identical harness settings across all evaluated models.
• Publishing a static leaderboard and then letting it go stale as authors move to the next paper. Either assign ongoing maintenance ownership or architect an unsaturatable (PvP) format from the start.
• Making agent exams too hard (no agent finishes, no signal) or too easy (all agents score >90%, no differentiation). Calibrate on the Difficulty Spectrum before public launch.
• Assuming AI researchers are sufficient to cover all important capabilities. The world's most important domain knowledge lives in practitioners who are not AI researchers — wastewater engineers, medical specialists, tradespeople. Failing to recruit them produces cognitively jagged AI.
• Ignoring compute costs in benchmark design. Running PvP games at statistical significance can require hundreds of thousands of game instances. Design with Bradley-Terry pairwise scheduling and cost ceilings before starting.
• Conflating model benchmarks with agent benchmarks. What gets wetter as it dries (a towel) is a model task. An agent completing a multi-step workflow involving tools, memory, and external APIs requires a completely different assertion and harness design.
• Relying on human expert judgment without planning for inter-expert alignment — even domain experts disagree, and AI cannot reliably judge innovation or creativity. Build explicit alignment workflows for human review stages.

// What are the key terms in the Agentic Evals framework?

Game Arena

A PvP benchmarking platform where AI models play games against each other with ELO/Bradley-Terry scoring. Inherently unsaturatable because there is always a winner and a loser, unlike static benchmarks that models eventually max out.

Standardized Agent Exams

A one-line prompt interface that allows any agent developer to submit their agent, have it take a standardized exam, and receive a score on a public leaderboard — democratizing eval access for consumer and low-resource agent builders.

Benchmark (Kaggle Benchmarks product)

A platform enabling any community member to build, run, and share evaluations openly. Structured as: Assertions → Tasks → Benchmarks, with LLM-as-judge and hard-coded checks combined.

Proprietary Novel Data Set

A benchmark dataset created from deep domain expertise that does not exist anywhere on the web and is not covered by AI lab research because it lacks immediate economic productivity — e.g., a wastewater treatment safety protocol benchmark built by a 20-year industry veteran.

Hill-climbing

The process of iteratively improving model performance by measuring against a benchmark. If something is not benchmarked, you cannot hill-climb on it — the capability will not improve systematically.

Cognitive Jaggedness

The uneven capability profile of AI models — superhuman in benchmarked areas, mediocre or untested in areas that lack evaluations. Caused by the Democratization Problem where a small number of researchers create all evals.

Democratization Problem

The structural imbalance where ~30,000 AI researchers create nearly all benchmarks for a world of 30M+ technical professionals and billions of end users, leaving vast swathes of human knowledge unevaluated.

Harness

The execution environment, scaffolding, tooling, and prompt structure surrounding a model during evaluation. The harness can account for 22%+ performance differences on identical tasks — making it critical to specify and control when comparing models.

Difficulty Spectrum

The calibration axis for benchmarks: too hard means no agent completes the task (no signal); too easy means no differentiation between agents (no signal). Useful benchmarks occupy the meaningful middle zone.

Bradley-Terry Pairing

A pairwise comparison scheduling method used in Game Arena to maximize statistical significance of ELO rankings while minimizing the number of games (and thus API cost) required to run.

LLM Model Proxy

A consistent interface layer that routes requests to multiple different model APIs in a standardized way, ensuring all models in a benchmark run are called under identical conditions.

Saturation

The state where models reach ceiling performance on a benchmark, eliminating its usefulness as a signal. Static benchmarks inevitably saturate; PvP Game Arena architectures are designed to be permanently unsaturatable.

// FREQUENTLY ASKED QUESTIONS