Frequently Asked Questions About Hetzel Agent Team Composition Framework
20 answers covering everything from basics to advanced usage.
// Basics
What does 'the model is already built' actually mean for my team?
It means the entire upstream pipeline — ingestion, training, cross-validation, and deployment — has already been done by providers like Anthropic and OpenAI, and exposed as an API endpoint. Your team should stop recreating that workflow and focus on everything downstream of the API: prompts, context, orchestration, evals, and observability.
What are the three role types this framework requires?
Data scientists / ML engineers; product, application, and systems engineers; and non-technical domain experts, subject matter experts, or product managers. A healthy agent team has all three. A team that is 100% data scientists or 100% engineers is a red flag because it will miss either proximity to the problem or eval rigour.
What is 'proximity to the problem' and why does it matter?
Proximity to the problem is how well a team member understands what the agent is actually meant to solve and how real users behave. The person with the closest proximity — often a domain expert or PM, not the ML engineer — holds disproportionate value because they can write high-quality context and judge whether the agent is behaving correctly and why.
What's the single most common mistake this framework prevents?
The isolation mistake — handing agent development entirely to the ML or data science team because generative AI has 'AI' in the name. This ignores systems engineering complexity and the proximity-to-problem advantage of domain experts, producing agents that impress internally but fail to reach production quality.
// How To
How do I audit an existing agent team that's stuck in POC mode?
Classify the organisation, then check for all three role types. Teams stuck in POC usually lack observability, treat fine-tuning as default, or gate prompt engineering behind technical staff. Confirm domain experts control context, engineers own the eval and observability pipeline, and data scientists validate the LLM judge against labelled data before blaming the model.
How do I assign data scientists on an agent team correctly?
Assign them to guardrails and eval validation, not systems or prompt engineering. Their job is to be the adult in the room on LLM risk, remind the team the model is just predicting tokens, create labelled datasets, and use precision/recall/F1 to confirm the LLM-as-judge actually agrees with human judgement.
How do I bring non-technical domain experts into the build?
Give them direct control over, or significant input into, the prompts and context seeded into the agent, and make them the primary human annotators reviewing agent traces. Do not gate this behind a technical intermediary — they judge both whether the agent performs well and, critically, why, catching edge cases engineers never anticipate.
How do I decide whether my use case needs fine-tuning?
Treat fine-tuning as the exception, not the rule. Before allocating data science resource, try to solve the problem with context engineering alone — most use cases can be. Reserve fine-tuning of open-source models as a deliberate, scoped assignment for cases that genuinely can't be solved through prompts and context.
// Troubleshooting
Why does my agent pass internal tests but degrade with real users?
You likely built strong evals but no observability. Evals build confidence during experimentation before production; observability maintains it once real users arrive. Without a production monitoring loop, confidence degrades rapidly. Close the loop by continuously harvesting production traces into your offline eval dataset and tracking LLM-judge alignment with human annotations.
Our LLM-as-judge scores look great but users complain — what's wrong?
Your judge may not be aligned with human judgement. LLM-as-judge is itself just a prompt and a model and can drift. Fix it by creating a labelled dataset from real interactions and applying recall/precision/F1 to measure whether the judge actually agrees with human reviewers. That validation is exactly where data scientists add unique value.
My team is all engineers with no data scientist — is that a problem?
It can be. Without a guardrails role, no one stress-tests the LLM's statistical limitations or validates eval quality, so the team may over-trust outputs. If you're AI Native and can't hire full-time, bring in a fractional data scientist or assign one engineer to own the guardrails and eval-validation function explicitly.
// Comparisons
How does this framework compare to a generic 'AI center of excellence' model?
A generic AI center of excellence tends to centralise agent work inside a specialist ML team — precisely the isolation mistake. The Hetzel framework instead distributes ownership by responsibility, keeping domain experts and systems engineers as first-class contributors. It optimises for proximity to the problem and production readiness rather than organisational tidiness.
How does agentic AI evaluation differ from traditional ML evaluation?
Traditional ML evaluation measures a narrow classifier on precision, recall, and F1. Agentic evaluation must assess functional performance across the full agent trace — did the agent do the right thing end-to-end for a real user. The traditional metrics still apply, but only to validate the evaluators themselves, not the agent's behaviour.
How is context engineering different from feature engineering?
Feature engineering changes model behaviour by transforming input data and retraining; context engineering changes an already-trained model's behaviour by editing prompts, context, and instructions. Feature engineering is a data science task; context engineering is often best done by non-technical domain experts because it requires deep understanding of user intent rather than statistical modelling.
Traditional Enterprise vs AI Native — how does org type change my team design?
Traditional Enterprises usually have a legacy ML platform team that inherits agent work top-down, making the isolation mistake likely; their fix is deliberately adding systems engineers and domain experts. AI Native companies have small cross-functional teams with high per-person proximity but often lack a guardrails role, so their fix is adding data science rigour.
// Advanced
How do I design the evals plus observability feedback loop?
Use evals during experimentation to build confidence before production, and observability afterward to maintain it. Close the loop by continuously harvesting production traces into your offline eval dataset and adding human-labelled (grounded) data over time, so LLM-judge alignment with human agreement can be tracked and self-corrected. Assign systems engineers to implement this pipeline.
Who should own distributed sub-agent orchestration?
Product and application engineers should own it. Supervisor agents calling child agents across different infrastructure and downstream systems is a complex systems problem, not a statistics problem. Assigning it to data scientists is a misallocation; the same engineers should also implement the eval and observability pipeline that closes the production-to-experimentation feedback loop.
What functional performance criteria should I define for an agent?
Define what the agent must actually do correctly, end-to-end, for a real user — not abstract accuracy. Have domain experts author these criteria because they hold proximity to the problem, then have data scientists validate the eval mechanism used to measure them. Only after this should you re-check whether your team composition matches those criteria.
How early should observability be implemented in an agent project?
From day one, especially in Traditional Enterprises. Implementing observability early lets production traces be human-annotated by domain experts as soon as real usage begins, feeding grounded data back into evals. Waiting until after launch means confidence built in experimentation silently degrades before you have the tooling to detect it.
Is an Agent Quality Platform necessary to apply this framework?
Not strictly, but it operationalises the two pillars the framework depends on — evals and observability. Platforms like Braintrust make it practical to run experiment-time evals, monitor production, and continuously feed labelled data back into the loop. Without such tooling, teams often build strong evals but neglect observability, undermining production reliability.