How to Standardize Agent Evals Across Your Org
For ML platform leads standardizing evals across teams · Based on Hetzel Eval Maturity Phases Framework
// TL;DR
If you're an ML platform lead responsible for eval standards across multiple teams, the Hetzel Eval Maturity Phases Framework gives you a shared vocabulary and a maturity continuum to assess where each team sits and what they need next. Use it to define what 'production-ready' means, mandate documented human annotation before automation, require teams to eval their LLM-as-judges against ground truth, and build shared platform tooling for full-trace ingestion and the production flywheel. It turns ad-hoc, inconsistent eval practices into a governed, scalable capability.
Why do you need a shared eval maturity model?
When every team invents its own eval approach, you get inconsistent quality bars, duplicated tooling, and agents that reach production without defensible evidence of quality. The Hetzel Eval Maturity Phases Framework gives you a shared vocabulary — vibe checking, measuring to manage, accounting for complexity, advanced techniques — so you can assess where each team sits and prescribe the right next action. Every eval decision maps back to one north star: agent quality, managing reputational, cost, and compliance risk.
How do you assess where each team is?
Map each team to one of the four maturity phases. A team only doing informal review sits at Level 1 and needs to add documented human annotation. A team with scoring functions but only synthetic datasets is at Level 2 and needs real production traces. Teams with tool-calling agents and CRUD complexity are at Level 3. Teams ready to scale failure discovery through topic modelling sit at Level 4. This gives you a portfolio view of eval maturity across the org.
What standards should you mandate first?
Start with two non-negotiables. First, documented human annotation with justifications, not just thumbs up/down — because human annotators hold domain knowledge you must extract before automating. Second, eval the eval: no team ships an LLM-as-judge into production scoring without validating it against a human-labelled ground truth dataset. Putting a robe and cloak on an LLM does not make it trustworthy. These two standards prevent the most common and dangerous failures.
How do you avoid teams treating evals like unit tests?
Educate teams that evals are not unit tests. Trying to exhaustively cover every failure path is infinite and unproductive — teams will spend all their time writing tests and none shipping. Instead, mandate that every eval effort starts high-level with known failure modes identified by a subject matter expert. Reinforce that 100% accuracy is not the goal; directional improvement is fine as long as scores trend correctly as teams iterate.
What platform tooling should you build or buy?
Several capabilities are shared across teams and belong in your platform. You need trace ingestion that can handle arbitrarily large multi-step traces so teams can score individual tool and MCP calls, not just final outputs. You need dataset management that ingests production and UAT traces. You need mock API and state-replay infrastructure so CRUD-based agents never write to production during evals. And at Level 4, you need CLI tooling for automated eval pipeline execution and topic modelling across production trace volume.
How do you make the flywheel an org-wide habit?
Institutionalise the flywheel: capture production traces, surface failures via human or automated tooling, pull failing examples into offline experimentation, rerun evals, and use results to guide the next improvement. Make this the default loop every team runs. It shifts your org from defensive evals — reacting to regressions — to playing offense, where eval results actively drive each agent improvement and measure the impact of every change.
Next step
Run a maturity assessment across your teams this quarter. Score each agent-owning team against the four phases, identify the single highest-leverage gap per team, and publish two org-wide standards — documented annotation with justifications, and eval-the-eval — before rolling out shared platform tooling.
// FREQUENTLY ASKED QUESTIONS
How do I define 'production-ready' for agents across teams?
Anchor 'production-ready' to the maturity phases and agent quality as the north star. At minimum require documented human annotation with justifications, validated scoring functions, a dataset built from real production or UAT traces, and any LLM-as-judge validated against human ground truth. Teams with tool calls must also demonstrate proper external state handling.
What's the highest-risk eval mistake to prevent org-wide?
Trusting LLM-as-judge outputs without evaling the eval. Judges are not inherently trustworthy, and if teams score at scale without validating against human ground truth, they build false confidence in agent quality. Mandate ground-truth validation as a hard gate before any judge scores production traffic.
Should every team reach Level 4?
No. The right maturity level depends on agent complexity and risk. A simple single-step agent may never need topic modelling at production scale. Use the framework to prescribe the appropriate next action per team, not to push everyone to Level 4. Maturity should serve agent quality, not be pursued for its own sake.