Frequently Asked Questions About Hetzel Eval Maturity Phases Framework
20 answers covering everything from basics to advanced usage.
// Basics
What does 'vibe checking' actually mean in evals?
Vibe checking is the earliest-stage eval practice: informally reviewing agent outputs without structured scoring. It's a legitimate starting point as long as it's paired with documented human annotation — recording a verdict and a written justification for each output. Vibe checking alone is fragile, but combined with documented annotation it becomes the raw material for deriving failure modes.
What are failure modes and why do evals target them?
Failure modes are the specific, identified ways an agent can produce bad outputs — hallucinations, unsafe outputs, excessive cost, compliance risks, format errors. Evals are built around failure modes rather than exhaustive hypothetical scenarios because covering every possible failure path is infinite and unproductive. Starting high-level with known failure modes, identified with a subject matter expert, keeps your eval effort focused and shippable.
What is the difference between a task, a dataset, and a scoring function?
The task is the agent or prompt under test. The dataset is the collection of example inputs used to initiate the task during an eval run — ideally populated with production or UAT traces. Scoring functions judge the utility or quality of each output, and can be deterministic (code-based) or non-deterministic (LLM-as-judge). Together they form a single eval run.
What does 'eval the eval' mean?
'Eval the eval' means validating your LLM-as-judge scoring functions by running them against a human-labelled ground truth dataset. It's required because putting a robe and cloak on an LLM doesn't make it inherently trustworthy. You measure the judge's alignment with human expert decisions before trusting it to score at scale — treating the judge itself as something that must be evaluated.
Why do evals exist at all — what's the north star?
Everything in the eval system exists wholly in service to agent quality: managing reputational risk, systems cost risk, compliance risk, and enabling confident iteration. Agent quality means the agent does what you expect when confronted with real usage and real users. Keep this north star in mind when deciding what to measure — if a metric doesn't serve agent quality, question why you're tracking it.
// How To
How do I derive failure modes from human annotations?
Feed the collected thumbs-down justifications into a coding assistant like Cursor, Claude Code, or Codex, and ask it to systematically extract and categorise the failure modes embedded in that language. The output is a structured list of failure modes your agent actually exhibits, drawn from real human judgement rather than guesswork. This bridges Level 1 vibe checking into Level 2 measuring to manage.
How do I decide between a code-based scorer and an LLM-as-judge?
For each failure mode, ask whether it can be caught deterministically — like too many tool calls, excessive token usage, or format errors. If yes, write a code-based scoring function. If the failure mode is subjective or nuanced, implement an LLM-as-judge using the human justification language from your annotations as the basis for the judge prompt. Then eval that judge against human ground truth.
How do I build an eval dataset from production traces?
Stop using only hand-crafted synthetic examples. Capture real production traces or UAT-level traces and add them to your dataset. Each entry should contain the inputs that initiate the task plus enough context to represent the system state at the time the trace was created. Think of the dataset as a snapshot of production, not an abstract test suite.
How do I evaluate a multi-step agent, not just its final output?
Instrument and capture the entire trace — every tool call and every intermediate step. Target scoring functions at individual steps, including individual tool or MCP calls, because each is a potential vector for failure. You'll need platform tooling capable of ingesting and querying arbitrarily large traces. Scoring only the final response misses failures that occurred earlier in the chain.
// Troubleshooting
My eval runs keep creating real data in production. How do I fix this?
This is a Level 3 CRUD-based tool call problem. Set up a mock API that accepts calls without writing real data, embed the relevant external system state directly into the captured trace payload, and configure your eval task to read that embedded state instead of querying the live system. If the API supports timestamp or version queries, replay the state as it existed when the original trace was captured.
My LLM-as-judge scores don't seem reliable. What's wrong?
You're likely trusting the judge without evaling the eval. Build a human-labelled ground truth dataset of agent outputs and measure how well your judge aligns with those human decisions. If alignment is poor, refine the judge prompt using the actual justification language from your annotations. Never trust judge scores at scale until they demonstrate acceptable agreement with a human expert.
I'm spending all my time writing evals and none shipping. What went wrong?
You're treating evals like unit tests — trying to exhaustively cover every possible failure scenario. Evals are not unit tests. Start high-level with the known, high-priority failure modes identified by a subject matter expert, and build evals only around those. Exhaustive coverage is infinite and unproductive. Focus your effort where the real risk lives.
My eval scores aren't hitting 100%. Is that a problem?
No — especially with LLM-as-judge, 100% accuracy is not the goal. Results can be directional. As long as scores trend in the right direction as you iterate on your agent, that's completely fine. Demanding perfect scores misunderstands the purpose of evals, which is to give you confidence and guide improvement, not to enforce a binary pass gate.
// Comparisons
How does this framework compare to a generic 'just test your prompts' approach?
Generic prompt testing is usually ad-hoc, unstructured, and stalls in proof-of-concept. This framework gives you a maturity continuum with concrete actions at each stage, insists on extracting domain knowledge before automating, validates your judges against ground truth, and closes the loop with a production flywheel. The result is measurable, defensible quality that bridges to production rather than vague confidence.
How is the eval flywheel different from just catching regressions?
Catching regressions is defensive — you're only reacting when something breaks. The flywheel plays offense: you proactively pull production failures into your offline eval dataset, rerun evals, and use results to guide each incremental improvement while measuring the impact of every change. It turns your eval system from a safety net into an engine for continuous agent quality improvement.
How do context-gathering tools differ from CRUD-based tools in evals?
Context-gathering tools read data and inject it into the LLM context without modifying external systems — they're lower risk in eval environments. CRUD-based tools create, read, update, or delete data in external systems, making them high risk because they can corrupt production data. CRUD tools require mocks or state isolation during eval runs; context-gathering tools mostly need accurate state representation in the trace.
// Advanced
What are advanced eval techniques at Level 4?
At Level 4, rather than relying solely on manual failure mode identification, you run topic modelling across production traces at scale to automatically surface emerging failure modes you didn't anticipate. You combine this with automated eval pipeline execution via CLI tooling so evals run in a fully automated, continuous manner. This scales failure discovery beyond what human review can reach.
How do I replay external system state that changed since the trace was captured?
For systems that support it — like some vector databases — use timestamp or version queries to replay the exact state that existed when the original production trace was captured. Where full state replay isn't possible, use mock APIs to approximate the real production environment. Either way, the goal is to represent the system state inside the trace so your eval reflects real conditions.
Can I skip vibe checking and go straight to LLM-as-judge?
Not effectively. Human annotators hold critical domain-specific knowledge about what quality looks like, and you must extract that knowledge — through justifications, not just thumbs up/down — before scaling it through LLM-as-judge. Skipping annotation means your judge prompt has no grounding in real expert reasoning, so its scores won't reflect what a human expert would decide.
What tooling do I need to run evals at the full trace level?
You need platform tooling capable of ingesting and querying arbitrarily large traces, since complex multi-step agents produce big traces spanning many tool calls and intermediate steps. At Level 4 you also want CLI tooling for automated eval pipeline execution and the ability to run topic modelling across production trace volume. Braintrust is the platform associated with this framework's originator.