Frequently Asked Questions About Hetzel Agent Team Composition Framework
21 answers covering everything from basics to advanced usage.
// Basics
What does 'the model is already built' actually mean?
It means the hard, expensive work of training and testing a foundation model has already been done by Anthropic, OpenAI, Mistral, and others. Your team's job is not to build a model — it's to implement, evaluate, and contextualise one. This fundamentally changes which skills matter most: implementation, context engineering, and evaluation rise in value while model training drops out.
What does 'proximity to the problem' mean in agent development?
Proximity to the problem is the degree to which a team member understands what the agent is actually meant to solve. Domain experts and subject matter experts typically have the highest proximity, which leads to better context engineering and human annotation quality. The people closest to the problem hold disproportionate value in agentic development — often more than the most technically skilled engineers.
Why are LLMs 'just APIs' from a product engineering view?
From a product engineering perspective, working with an LLM means sending a payload and receiving a response, then making it useful to the end user. This is exactly the pattern product and application engineers have deep experience with. Reframing LLMs as APIs opens agent development to a much broader group than traditional ML specialists — you don't need a PhD to integrate an API.
What are the three personas in the Hetzel framework?
The three personas are: (1) Data Scientists / ML Engineers, who handle guardrails, statistical literacy, eval validation, and any fine-tuning; (2) Product / Application / Systems Engineers, who handle implementation, infrastructure, and observability pipelines; and (3) Non-Technical Domain Experts, who lead context engineering and human annotation. Each contributes irreplaceable value at different stages — the answer is always in the middle.
// How To
How do I audit my current team for coverage gaps?
Map every current team member against the three personas: data scientists/ML engineers, product/application/systems engineers, and non-technical domain experts. Identify which personas are missing or marginalised. A team staffed entirely by ML engineers is a warning sign of a coverage gap. Fix gaps before building begins — especially the absence of a domain expert, which is the most damaging.
How do I redirect ML engineers away from the wrong metrics?
Reassign your ML engineers to three agent-specific roles: (a) act as the 'adult in the room' on LLM risk and statistical literacy, (b) validate LLM-as-judge evals against labelled datasets using recall, precision, and F1, and (c) lead fine-tuning of open-source models only if the use case genuinely requires it. Redirect them away from treating traditional ML metrics as the primary eval signal for agent behaviour.
How do I involve non-technical domain experts in building agents?
Give domain experts meaningful control over prompt and context engineering — the primary lever for changing agent behaviour — and deploy them in human annotation workflows. They should review agent traces and label whether the agent performed well or poorly, and critically, explain why. Do not treat this as optional or cosmetic; their proximity to the problem makes their input central to agent quality.
How do I set up an eval and observability pipeline?
Have the whole team agree on what 'good' looks like functionally, not just technically. Build evals for pre-production experimentation and observability for post-production monitoring. Use production data to continuously expand your offline evaluation dataset. Assign product/systems engineers to build the pipeline, data scientists to validate LLM-as-judge evals against human labels, and domain experts to annotate traces. Check whether judge evals converge toward human agreement over time.
// Troubleshooting
My agent worked in testing but fails in production — why?
Confidence built in experimentation does not transfer automatically to production. Real usage confronts the agent with scenarios your evals never anticipated. If you skipped the observability pipeline, you have no way to catch these failures or feed them back into your offline eval dataset. Add post-production monitoring so real behaviour continuously expands your evaluation coverage.
My LLM-as-judge evals seem unreliable — what's wrong?
LLM-as-judge evaluators are just prompts and models, so they can drift from human agreement over time without a self-check mechanism. Validate them against human-labelled ground truth datasets using precision, recall, and F1. Track whether the judge is converging toward or diverging from human agreement. If they run unchecked, you're trusting an unvalidated evaluator — a common and dangerous pitfall.
Our agent lacks relevance and context — how do we fix it?
This usually means your team has no one with high proximity to the problem — likely a team of only engineers with no domain expert. Pressure-test the team against the 'Proximity to the Problem' principle: does at least one person deeply understand what the agent is meant to solve? If not, bring in a domain expert to lead context engineering before building further, not after.
We handed agents to our ML platform team by default — is that a problem?
Likely yes — this is the most common Traditional Enterprise mistake, driven by 'it has AI in the name'. It leads to teams optimising for the wrong metrics and excluding domain experts and product engineers. Audit the team for coverage gaps, redirect ML engineers to guardrail and eval validation roles, add product engineers for integration and infrastructure, and recruit domain SMEs for context engineering and annotation.
// Comparisons
How does the Hetzel framework compare to a generic 'hire ML engineers' approach?
The generic approach assumes agent building is an ML problem, so you hire ML engineers. The Hetzel framework rejects this: the model is already built, so ML skills alone leave you optimising the wrong metrics with no domain grounding. It insists on a deliberately diverse team spanning data scientists, product engineers, and domain experts — because no single discipline owns agents.
How does context engineering compare to fine-tuning?
Context engineering changes agent behaviour by adjusting prompts and inputs to a pre-built LLM — fast, cheap, and accessible to domain experts. Fine-tuning retrains an open-source model — slow, expensive, and rare. Context engineering is the default lever and should handle most behaviour changes; fine-tuning is reserved for cases that genuinely demand it. Treating fine-tuning as the default is a well-documented pitfall.
How does functional performance compare to technical metrics like F1?
Functional performance measures whether the agent actually accomplishes its purpose for real users — does it resolve the customer's query correctly and safely? Technical metrics like precision, recall, and F1 measure narrow classification accuracy suited to a two-box ML pipeline. Agents have a far broader surface area, so locking onto F1 alone is a trap. Functional performance should be your primary eval signal.
How does agent evaluation differ from traditional ML cross-validation and A/B testing?
Traditional ML uses cross-validation and A/B testing on a model you built. Agent evaluation covers a far broader functional surface area on a model you didn't build. The pipeline is entirely different: you use evals for pre-production confidence, observability for post-production monitoring, LLM-as-judge validated against human labels, and human annotation of traces. Applying the traditional cross-validation dance to agents is a pitfall.
// Advanced
How do I staff a distributed multi-agent system differently?
Distributed architectures — a supervisor agent orchestrating sub-agents on different infrastructure and calling different systems — are a complex systems engineering problem. Weight your team toward strong systems and application engineers who can manage the infrastructure where agents execute, plus robust observability across all components. Still include domain experts for context engineering and data scientists for eval validation, but recognise infrastructure complexity rises sharply.
When is fine-tuning an open-source model actually justified?
Fine-tuning is justified only when the use case genuinely demands it — for example, highly specialised domains where context engineering alone cannot achieve required behaviour, or when latency, cost, or data-privacy constraints require an in-house open-source model. Even then, assign your data scientists to it. For the vast majority of agent use cases, context engineering with pre-built LLM APIs is sufficient and far more efficient.
How do I keep my offline eval dataset from going stale?
Use production data to continuously expand your offline evaluation dataset. Your observability pipeline should surface real scenarios the agent encounters — including failures your original evals never anticipated — and feed those back into offline evals. This closes the feedback loop between production and experimentation, ensuring your eval coverage grows to match how the agent is actually used over time.
What's the ideal balance between rigour and speed for an AI Native team?
AI Native teams already have proximity to the problem and move fast, but risk under-engineering rigour and guardrails. Add a data scientist or someone with a stats background to build guardrails and design LLM-as-judge eval pipelines validated against labelled data. Formalise human annotation with a domain expert. Build an observability pipeline. The goal is layering rigour onto speed without killing the agility that makes AI Natives effective.
Who should own the definition of 'good' for an agent?
No single person — the whole team must agree jointly on what 'good' looks like functionally, not just technically. Domain experts define whether the agent solves the real problem, data scientists ensure eval validity and guardrails, and product engineers build the pipeline that measures it. This shared definition prevents the classic failure of engineers optimising metrics that don't reflect real user value.