Why PMs Are Central to Agent Quality
For Product managers on AI teams · Based on Hetzel Agent Team Composition Framework
// TL;DR
Product managers and domain experts are often treated as bystanders on agent teams — but the Hetzel framework puts them at the center. Because the model is already built, agent behaviour changes by changing inputs, and the person with the closest proximity to the problem writes the best inputs. That's frequently the PM, not the ML engineer. Use this framework to claim direct ownership of prompts and context, become the primary human annotator of agent traces, and define functional success criteria — the responsibilities that most determine whether an agent reaches production quality.
Why do you hold more leverage than the ML engineer?
Because of proximity to the problem. The Hetzel framework's core insight is that the person who best understands what the agent is meant to solve and how real users behave holds disproportionate value — and that person is often the PM or domain expert, not the ML engineer. Since the model is already built, behaviour is changed by editing prompts, context, and instructions, and writing good inputs requires understanding user intent, not statistics.
This is context engineering, the generative AI analogue to feature engineering. It's the highest-leverage skill on the team, and it can be performed by non-technical experts. That's you.
What should you directly own?
Three things. First, the prompts and context seeded into the agent — you should have direct control or significant input, not have your changes gated behind a technical intermediary. Second, human annotation: you should be a primary reviewer of agent traces, labelling whether the agent performed well and, crucially, why. Third, functional success criteria — defining what the agent must actually do correctly, end-to-end, for a real user.
That last point matters because teams often let precision, recall, and F1 stand in as agent success metrics. Those measure classifiers and evaluators, not end-to-end agent behaviour. You're the person who can define real functional performance, and the framework explicitly assigns that responsibility to domain experts.
How do you work alongside data scientists and engineers?
Divide by strength. Data scientists own guardrails — reminding the team the model is just predicting tokens, and validating whether your LLM-as-judge evaluator actually agrees with human labels using precision/recall/F1. Product and systems engineers own API integration, sub-agent orchestration, and the eval-plus-observability pipeline. You own the human and functional layer: what good looks like, why, and the annotated data that grounds everything.
Your annotations are the fuel. The evals-plus-observability loop closes when production traces are human-labelled and fed back into the offline eval dataset — and your labels are what let the team track whether the LLM judge stays aligned with human judgement over time.
How do you spot a team that's about to fail?
Watch for the isolation mistake: agent development handed entirely to the ML team because 'AI' is in the name. If that's happening, no one holds proximity to the problem and no one is doing real context engineering — the agent will impress in demos and fail with real users. Also watch for strong evals but no observability, which produces confidence that evaporates the moment the agent meets production traffic.
If you're being kept out of prompt and context decisions, that's a red flag too. The framework is explicit: don't gate context engineering behind technical staff. Push to be in the room and, ideally, in the editing interface.
What does production readiness look like from your seat?
It means functional criteria are defined and owned by you, prompts and context are being iterated by people close to the problem, data scientists have validated the eval mechanism, and observability is live so real traces flow back into evals. When those pieces are in place, you've moved from a prolific POC team to one that ships reliably.
Next step: Ask for direct edit access to your agent's prompts and context this week, volunteer as a primary trace annotator, and draft the functional success criteria that define what 'good' means for your users.
// FREQUENTLY ASKED QUESTIONS
I'm not technical — should I really be editing prompts?
Yes. The framework is explicit that prompt and context engineering should not be gated behind technical staff. As the person with the closest proximity to the problem, you understand user intent and edge cases better than engineers do. Editing prompts is context engineering — the highest-leverage skill on an agent team — and it doesn't require coding.
How do I convince engineers to let me own context engineering?
Point to proximity to the problem: behaviour in agentic AI changes by changing inputs, and the best inputs come from whoever understands real users. Offer to be the primary human annotator of agent traces too, since your labels ground the eval dataset. Framing your role as fueling the evals-plus-observability loop makes the value concrete to engineers.
What's the difference between what I define and what data scientists validate?
You define functional success criteria — what the agent must do correctly end-to-end for a real user. Data scientists validate the eval mechanism that measures those criteria, using labelled data and precision/recall/F1 to confirm the LLM-as-judge agrees with human reviewers. You set the target; they verify the instrument measuring it is trustworthy.