How Should ML Leads Staff Agent Teams?
For Enterprise ML/Data Science leads · Based on Hetzel Agent Team Composition Framework
// TL;DR
If your ML or data science team was handed agent-building by default because 'it has AI in the name', the Hetzel Agent Team Composition Framework helps you avoid the classic Traditional Enterprise trap. It shows why a team of only ML engineers optimises for the wrong metrics and lacks proximity to the problem. Instead, redirect your ML engineers to guardrails and eval validation, add product engineers for LLM-as-API integration and infrastructure, and recruit domain SMEs to lead context engineering and human annotation. The result is an agent grounded in the real problem, measured on functional performance rather than F1 alone.
Why did my ML team get handed agent development?
Most likely because your organisation categorised generative AI as an 'AI problem' and delegated it top-down — the most common Traditional Enterprise mistake. The logic feels intuitive: agents involve AI, your team does AI, so your team should build agents. But the Hetzel framework exposes the flaw. Traditional ML assumes you build and train a model. With agents, the model is already built by Anthropic, OpenAI, and Mistral. Your team's job is to implement, evaluate, and contextualise it — a fundamentally different skill set.
This means a team staffed entirely by ML engineers is a warning sign. It will over-index on precision, recall, and F1 — technical metrics for a two-box ML pipeline — and under-include the non-technical domain experts who hold proximity to the problem.
What role should my data scientists actually play?
Redirect them away from treating traditional ML metrics as the primary eval signal. Instead, assign three agent-specific roles:
1. The adult in the room on LLM risk and statistical literacy — you still need someone who understands variance, drift, and failure modes.
2. Eval validation — validate LLM-as-judge evaluations against human-labelled datasets using recall, precision, and F1. Here those metrics are useful: for checking whether your judge agrees with humans, not for scoring the agent directly.
3. Fine-tuning — but only if the use case genuinely demands it. Most agent behaviour changes through context engineering, not retraining. Treating fine-tuning as the default is a pitfall.
This reframe keeps your best statistical talent contributing where it's irreplaceable without misdirecting the whole team.
Who is missing from my team?
Audit your current composition against three personas. You almost certainly have data scientists and ML engineers. You are likely missing product/application/systems engineers — the people who treat LLMs as APIs, integrate them into the product, manage the infrastructure where agents execute, and build the eval and observability pipelines. And you are very likely missing non-technical domain experts — the customer service specialists, compliance officers, or subject matter experts with the highest proximity to the problem.
Consider the canonical example: a financial services firm assigns its ML platform team to build a customer-facing account-query agent. The fix is to redirect ML engineers to guardrail and eval roles, bring in product engineers for the LLM-as-API integration, and recruit customer service SMEs for context engineering and human annotation of agent traces. Then redefine eval criteria beyond precision/recall to include functional performance — does the agent actually resolve the query correctly and safely?
How do I measure agent quality without over-relying on F1?
Agent quality has two pillars: evals (pre-production experimentation) and observability (post-production monitoring). Define what 'good' looks like functionally as a whole team — not just technically. Use LLM-as-judge evals validated against human labels, and check whether the judge converges toward or diverges from human agreement over time. Deploy your domain experts in human annotation workflows to review traces and explain why the agent performed well or poorly. Then use production data to continuously expand your offline eval dataset, closing the feedback loop between production and experimentation.
Remember: confidence built in experimentation does not transfer automatically to production. Real usage confronts the agent with scenarios your evals never anticipated. Skipping observability is a serious pitfall.
Next step
Run the seven-step Hetzel workflow on your current team this week: classify your org type, audit for coverage gaps across the three personas, reassign your data scientists to agent-specific roles, and identify which product engineers and domain experts you need to recruit before building — not after.
// FREQUENTLY ASKED QUESTIONS
Should I refuse the agent mandate if my team is all ML engineers?
Don't refuse — reshape. A team of only ML engineers is a warning sign, not a dead end. Use the mandate to recruit the missing personas: product/application engineers for integration and infrastructure, and non-technical domain experts for context engineering and annotation. Redirect your ML engineers to guardrails, eval validation, and any genuinely necessary fine-tuning. The answer is always in the middle.
Are precision, recall, and F1 ever useful for agents?
Yes — but not as the primary signal for scoring agent behaviour. Use them to validate your LLM-as-judge evaluations against human-labelled datasets, checking whether the judge agrees with humans. Scoring the agent itself should centre on functional performance: does it actually accomplish its purpose for real users? Locking onto F1 as the main agent metric is a documented trap.
How do I convince leadership we need non-technical domain experts?
Frame it around proximity to the problem: the people closest to what the agent solves produce the best context engineering and annotation quality — the primary levers of agent quality. An agent built entirely by engineers with no domain expert will lack contextual grounding and be irrelevant to users. Fixing this before building is far cheaper than after.