Frequently Asked Questions About Kaggle DeepMind Agentic Evals at Scale Framework
21 answers covering everything from basics to advanced usage.
// Basics
What does 'agentic evaluation' actually mean?
Agentic evaluation tests an agent completing a multi-step workflow involving tools, memory, and external APIs — not a single model response. It differs fundamentally from model benchmarks like 'what gets wetter as it dries.' Agentic tasks require different assertion designs, a controlled harness, and awareness that the execution environment can drive 22%+ of performance differences.
What is a harness in AI evaluation?
A harness is the execution environment, tool scaffolding, and prompt structure surrounding a model during evaluation. It's critical because harness configuration alone can account for 22%+ performance differences on identical models and tasks. When comparing models, you must document and lock the harness so you're measuring the model, not accidentally measuring your setup.
What is cognitive jaggedness and why does it matter?
Cognitive jaggedness is the uneven capability profile of AI models — superhuman in benchmarked areas, mediocre or untested elsewhere. It's caused by the Democratization Problem, where a tiny group of researchers creates all evals. It matters because unevaluated capabilities never systematically improve, producing AI that fails unpredictably in domains no one benchmarked.
What is a Proprietary Novel Data Set?
A Proprietary Novel Data Set is a benchmark built from deep domain expertise that doesn't exist anywhere on the web and isn't economically productive for AI labs to pursue — like a 20-year wastewater engineer's safety protocols. These are the most valuable benchmarks precisely because no lab has the data and no scraped corpus captures the practitioner's tacit knowledge.
// How To
How do I structure assertions, tasks, and benchmarks?
Write assertions any third party can verify independently — combine hard-coded checks (does the output contain X?) with LLM-as-judge for qualitative dimensions. Group related assertions into Tasks, then group Tasks into Benchmarks. Expose all configuration publicly so no one can accuse you of optimizing for a specific model. This hierarchy keeps evals modular and reusable.
How do I run a Standardized Agent Exam for a consumer agent?
Pass a one-line system prompt describing your agent to the exam endpoint and receive a score on a public leaderboard. Run safety-focused tracks first — does the agent refuse to forward sensitive data? Does it respect scope boundaries? Then compare against the 500+ already-evaluated agents to contextualize performance without building any custom harness.
How do I use hackathons to build domain-specific evals?
Define guardrails (specific focus areas, e.g., five cognitive faculties) but give participants creative latitude. Provide free access to data hosting, API credits for state-of-the-art models, and writeup tools so work is reusable. Require all outputs to be open source. Hackathons convert the Democratization Problem into a strength by channeling diverse expertise at scale.
How do I document a harness so results are reproducible?
Use a consistent LLM model proxy layer so all models are called identically, then document: model API version, context window used, whether features like context compaction are enabled or disabled, temperature, and every other config. A domain expert's benchmark and a frontier lab's benchmark must run under identical harness conditions to be comparable.
// Troubleshooting
My benchmark saturated within months — what went wrong?
You likely used a static leaderboard for a finite task space that models climbed to ceiling. Either re-architect to a PvP Game Arena with ELO/Bradley-Terry scoring for permanent unsaturatability, or increase task difficulty into the meaningful middle of the Difficulty Spectrum. Static benchmarks inevitably saturate — plan for it from the start.
No agent can complete my benchmark tasks — how do I fix it?
Your benchmark is too hard and producing no signal. Run a pilot with 3-5 representative agents and reduce task complexity until you see meaningful spread — not all failing, not all passing above 90%. Also check whether long agentic tasks are hitting token or cost limits before the agent can finish, which mimics a difficulty problem.
My PvP evaluation is too expensive to run — what should I do?
Stop running full round-robins. Switch to Bradley-Terry pairwise scheduling, which selects the next matchup with the most information gain given current ELO uncertainty, dramatically reducing games needed. Set explicit cost ceilings before starting. Remember one poker eval required 400,000 hands at statistical significance — architecture choices directly control your compute bill.
Domain experts disagree on the correct answer — how do I handle it?
Build explicit inter-expert alignment workflows into your human review stages rather than assuming a single ground truth. Even domain experts disagree, and AI cannot reliably judge innovation or creativity. Plan for structured reconciliation, document disagreements, and where consensus is impossible, treat the dimension as qualitative signal rather than a hard-coded assertion.
// Comparisons
How does this framework compare to using SWE-Bench or MMLU directly?
Fixed benchmarks like SWE-Bench and MMLU are valuable but saturate and cover only what their authors chose. This framework is a methodology for building evals that stay live (via PvP), enforce harness transparency, and recruit domain experts to cover gaps those benchmarks miss. It complements standard benchmarks rather than replacing any single one.
How is LLM-as-judge different from hard-coded assertions?
Hard-coded assertions deterministically check verifiable facts (does the output contain the correct shutdown code?), while LLM-as-judge scores qualitative dimensions like reasoning quality or safety nuance that can't be pattern-matched. The framework combines both: use hard-coded checks for objective correctness and LLM judging for subjective reasoning, keeping everything transparent and reproducible.
How does a PvP Game Arena compare to an assertion-based benchmark?
PvP Game Arenas pit models against each other with ELO scoring and are permanently unsaturatable — ideal for finite, competitive task spaces. Assertion-based benchmarks check outputs against defined criteria and suit open-ended or proprietary domains. Choose PvP when saturation risk is high within 12-18 months; choose assertions when domain knowledge is the differentiator.
Why not just trust the benchmark charts model publishers release?
Because publishers can tune configurations, prompt strategies, and API features to favor their own model, then present results as neutral. This framework treats publisher charts as untrustworthy by default and requires exposing full configuration for third-party reproduction. Running identical harness settings across all models is the only way to get comparable, credible numbers.
// Advanced
How do I decide between evaluating the model, the agent, or the harness?
Decide by what you actually want to learn, then isolate it. If you want raw model capability, lock the agent and harness as constants. If you're comparing agent designs, lock the model and harness. Never conflate all three — a benchmark mixing them produces misleading results and can't attribute performance to any single component.
What emergent signals can PvP evaluations reveal beyond rankings?
Beyond ELO rankings, PvP evaluations surface emergent model personalities — such as risk-seeking versus risk-averse negotiation behaviors — that static benchmarks can't capture. Always track and publish full LLM conversation logs as a dataset so the community can analyze these secondary behaviors, deception strategies, and failure modes that only appear in adversarial, multi-turn play.
What reproducibility artifacts must I publish for a credible eval?
Publish the benchmark tasks, the full harness config, the raw LLM conversation logs, the ELO/score methodology, and a visualizer or qualitative examples where possible. Make everything forkable and open source. A leaderboard without these artifacts is marketing, not science — third parties must be able to independently reproduce every number you report.
How do I design an agentic benchmark that isolates a single capability?
Design a task or game that structurally requires the target capability and nothing else — for multi-step deception, use a negotiation game with hidden information. Control the harness so tool access and prompt structure don't leak advantages, calibrate difficulty for spread, and use LLM-as-judge plus hard-coded checks to score. Isolation is what makes the signal attributable.
Can I convert an existing static benchmark into an unsaturatable one?
Sometimes — if the underlying task can be reframed as competitive (model vs. model), you can wrap it in a PvP structure with ELO scoring for permanent unsaturatability. If it's inherently a single-output correctness check, you can't make it PvP, but you can assign ongoing maintenance ownership and periodically raise difficulty to keep signal alive.