How to Monitor a Clinical AI Agent Properly
For Healthcare AI product teams · Based on Hetzel Agent Observability Differentiation Framework
// TL;DR
Healthcare AI product teams building LLM-based triage or clinical assistants can't rely on Datadog alone — it captures latency and errors but never evaluates whether an agent's response was clinically grounded, recommended appropriate pathways, or used approved language. The Hetzel framework diagnoses this gap, brings clinicians into the observability workflow to grade traces with written justifications, and flags that large clinical trace payloads need purpose-built ingestion. Use it when your DevOps team assumes an existing monitoring contract covers your agent — because functional, clinical quality is a separate discipline requiring domain-expert review.
Why isn't Datadog enough for a clinical AI agent?
Your DevOps team has a Datadog contract and wants to point it at the new triage assistant. That instinct is half right. Datadog will capture latency, error rates, and uptime — real technical observability. But it cannot evaluate whether the assistant's responses were grounded in the patient's retrieved records, whether it recommended appropriate clinical pathways, or whether it deviated from approved clinical language.
This is the Scope Difference Principle in action. Traditional observability answers 'is the system up?' Agent observability must also answer 'was the clinical output correct?' Those are fundamentally different questions, and no amount of latency dashboards will surface a subtly wrong triage recommendation.
Who needs to review the agent's traces?
Not just engineers. The Dual Persona Requirement says registered nurses and clinicians must be included in the observability workflow. They are closest to the patients and the clinical problem space, and they can evaluate qualitative agent quality in natural language — something a systems engineer simply can't do for a medical recommendation.
Build a human annotation workflow where clinicians review production traces, assign quality grades, and — critically — write justifications for those grades. Don't skip the justifications. A grade of 'poor' tells you nothing scalable; a justification like 'the assistant recommended a pathway that ignored the documented allergy in the retrieved record' becomes the seed for an automated groundedness and protocol-adherence score.
How do I handle the massive size of clinical traces?
Expect trouble here. Clinical agent traces contain full conversation context plus retrieved medical documents, which makes them large and semi-structured — a textbook Agent Traces Are Nasty scenario. Individual spans can reach megabytes; full traces can be enormous.
Traditional observability databases weren't built for this. You'll need full-text indexing and write-ahead log ingestion to store, search, and stream these traces in real time. When a compliance officer asks 'show me every trace where the agent mentioned a specific medication,' you need full-text search across trace content to answer it — a read pattern standard backends don't support.
What failure modes should I automate versus discover?
Separate known unknowns from unknown unknowns. For known unknowns, define automated scores now: groundedness in the patient record, correct tool usage, adherence to approved clinical language. For unknown unknowns, run LLM-based embedding and clustering over production traces to surface emergent patterns — unexpected question types, sentiment signals, and failure modes you didn't anticipate during design.
Then close the iteration loop: when a clinician flags a bad trace, add it to an offline dataset, run evals in batch, and experiment with a fix. Because observability and evals are the same system, this handoff should be immediate — not a ticket that sits for weeks.
What should I do next?
Start by classifying your system's determinism profile and auditing your metric scope: list every clinical quality property you care about and check whether Datadog can measure it. The gaps you find are your agent observability requirements. Then recruit two or three clinicians into a weekly trace-review ritual with mandatory justifications — that's the single highest-leverage step to move from technical-only monitoring to real clinical quality assurance.
// FREQUENTLY ASKED QUESTIONS
Can Datadog evaluate whether a clinical AI response was correct?
No. Datadog captures latency, error rates, and uptime — technical observability only. It cannot evaluate whether the assistant's response was grounded in the patient's retrieved records, recommended an appropriate clinical pathway, or used approved clinical language. Those functional quality judgments require agent-specific observability and clinician review, not a traditional monitoring tool.
Do clinicians really need to review agent traces?
Yes. The Dual Persona Requirement states that registered nurses and clinicians must participate because they're closest to patients and can evaluate qualitative agent quality in natural language. Engineers can't judge whether a triage recommendation was clinically appropriate. Excluding clinicians because it 'feels too technical' means missing the exact signal that catches dangerous errors.
Why are clinical agent traces so large?
Clinical traces contain full conversation context plus retrieved medical documents, making them large and semi-structured — an Agent Traces Are Nasty scenario. Spans can reach megabytes and full traces can be enormous. This requires purpose-built ingestion with full-text indexing and write-ahead log support, since traditional observability databases weren't designed for this volume or structure.