Hetzel Agent Observability Differentiation Framework

Last updated: 29 May 2026

Accurately diagnose whether a given AI agent system requires traditional observability tooling, agent-specific observability, or both — and design the right observability stack accordingly.

// TL;DR

The Hetzel Agent Observability Differentiation Framework helps you diagnose whether an AI agent system needs traditional observability tooling (like Datadog or Grafana), agent-specific observability, or both — then design the right stack accordingly. Use it whenever you're building, auditing, or advising on observability for LLM-based agents, or when stakeholders ask why existing tools fall short for agent workloads. The framework separates technical observability (latency, tokens, errors) from functional observability (groundedness, tool use, response quality), and insists domain experts — not just engineers — participate in reviewing agent traces.

Framework

// When should you use the Hetzel Agent Observability Differentiation Framework?

Use this skill whenever you are designing, auditing, or advising on the observability strategy for an AI agent system. Also apply it when stakeholders ask why existing tools like Grafana or Datadog are insufficient for agent workloads.

// What information do you need before diagnosing your observability stack?

system_typerequired
Is the system a traditional deterministic application, an LLM-based agent, or a hybrid?
existing_toolingrequired
What observability tools are already in use (e.g., Datadog, Grafana, ClickHouse)?
stakeholder_personasrequired
Who needs to consume the observability data — systems engineers only, or also domain experts like clinicians, lawyers, wealth advisors?
agent_use_caserequired
What is the agent doing? What domain does it operate in? What does 'quality' mean for this agent?
scale_expectations
Expected volume of agent interactions in production — rough order of magnitude.

// What are the core principles behind agent observability differentiation?

Scope Difference Principle

Traditional observability is entirely about uptime and technical performance — is the application up, and is it delivering the expected user experience from a technical lens. Agent observability has a fundamentally different and broader scope: it must also measure qualitative, semantic, and reasoning-level properties of agent behavior.

Non-Determinism Problem

Traditional applications have deterministic code paths and known control flow. Agents are non-deterministic because LLMs have high variety and are abstracted. This means you cannot rely on constrained, known metrics alone — you must measure why an agent took one path versus another.

Agent Traces Are Nasty

Agent traces are highly semi-structured, contain massive volumes of unstructured text, can exceed a gigabyte per trace with individual spans reaching 20 megabytes, and must still be delivered in true real time. This is a completely new systems problem that traditional observability infrastructure was not designed to handle.

Functional vs. Technical Observability

Agent observability has two layers: functional observability (was the agent's output grounded, aligned to brand standard, using the right tools, producing quality responses?) and technical observability (latency, time to first token, token count, duration). Technical observability comes 'on the house' when you properly trace an agent; functional observability requires purpose-built tooling.

Dual Persona Requirement

Traditional observability is consumed exclusively by systems or product engineers. Agent observability, done well, must include both technical and non-technical people — domain experts like clinicians, lawyers, or wealth advisors who are closest to the users or the problem space and can evaluate qualitative agent quality in natural language.

Observability and Evals Are the Same Problem

Observability and evals are solved with the same underlying system. The only difference is that with evals, you know the inputs ahead of time and run them in batch; with observability, you do not know the inputs ahead of time and process them in real time.

Known Unknowns vs. Unknown Unknowns

Some agent quality issues are known unknowns — failure modes you can anticipate and build automated scores for. Others are unknown unknowns — patterns and issues you discover by running clustering and topic modeling over production traces. A complete agent observability strategy must address both.

Human Annotation as a Scaling Seed

Human annotation — having domain experts grade agent traces and justify their grades — is a critical step. Those justifications become the raw material for building scalable, automated scoring functions. You find the failure modes through human review first, then systematize them.

// How do you apply the agent observability framework step by step?

1
Classify the system's determinism profile
Ask: does this system follow known control flow with deterministic code paths, or does it use an LLM where the reasoning path is variable? If deterministic, traditional observability may be sufficient. If non-deterministic (agent), proceed to step 2.
2
Audit the scope of what needs to be measured
Separate technical metrics (latency, duration, time to first token, total tokens, error counts, cache hits) from functional/qualitative metrics (groundedness in retrieved context, correct tool usage, alignment to brand standard in system prompt, response quality). Traditional tools cover the former; agent observability tooling is required for the latter.
3
Assess the trace data characteristics
Determine expected trace size and structure. If traces are semi-structured, contain large volumes of unstructured text, or could exceed megabytes per span, flag this as a 'agent traces are nasty' scenario. Traditional observability databases (including OLAP tools like ClickHouse) are likely insufficient without augmentation for full-text indexing and write-ahead log ingestion.
4
Map the required read patterns
Identify whether consumers need: (a) real-time streaming visibility as interactions happen, (b) SQL/analytical queries over historical traces for experimentation, or (c) full-text search across trace content (e.g., 'show me every trace that mentioned the word X'). If all three are needed, a purpose-built agent trace database or stack is required, not a standard observability backend.
5
Identify all stakeholder personas who need access
List every role that must consume or act on observability data. If the list includes only systems/product engineers, traditional tooling personas apply. If it includes domain experts (clinicians, lawyers, wealth advisors, compliance officers), the platform must support natural language interaction and trace review by non-technical users.
6
Design the human annotation workflow
Establish a process where domain experts review production traces, assign quality grades, and — critically — provide written justifications for their grades. These justifications are the seed for building automated, scalable scoring functions. Do not skip justifications; grades alone are insufficient.
7
Separate known unknowns from unknown unknowns
For known unknowns: define explicit automated scores tied to anticipated failure modes (groundedness checks, tool-use checks, brand alignment checks). For unknown unknowns: implement lightweight LLM-based embedding and clustering over production traces to surface emergent topics, user intent patterns, sentiment signals, and unexpected failure modes you did not anticipate.
8
Close the iteration loop between production and experimentation
Once issues surface in production observability traces, the workflow should make it fast and direct to: (a) add those traces to an offline dataset, (b) run evals against them in batch, and (c) experiment with agent changes. The goal is to compress the time between 'problem seen in production' and 'fix validated in experimentation'.
9
Confirm whether traditional observability tools are still needed in parallel
Traditional tools like Datadog or Grafana are still valid and useful for the technical layer — 400/500 errors, uptime, website-level performance. Do not replace them; layer agent observability on top for functional quality. The two are complementary, not competitive, at this layer.

// What do real-world applications of this framework look like?

A healthcare company has built an LLM-based triage assistant and their existing DevOps team wants to monitor it using their current Datadog contract.

Apply the Scope Difference Principle: Datadog will capture latency, error rates, and uptime (technical observability) but cannot evaluate whether the assistant's responses were grounded in the patient's retrieved records, whether it recommended appropriate clinical pathways, or whether it deviated from approved clinical language. Invoke the Dual Persona Requirement: registered nurses and clinicians must be included in the observability workflow to grade traces and provide justifications. Use the Human Annotation as a Scaling Seed principle to turn their justifications into automated scoring functions for clinical accuracy and protocol adherence. Flag Agent Traces Are Nasty: trace payloads containing full conversation context and retrieved medical documents will be large and semi-structured, requiring a purpose-built ingestion and indexing strategy.

A fintech team is building a wealth management agent and wants to understand how users are actually using it in production — they don't know what questions to ask yet.

This is an unknown unknowns problem. Traditional observability will show you uptime and error rates but will not reveal user intent, sentiment, or emergent usage patterns. Apply the unknown unknowns branch of the Known Unknowns vs. Unknown Unknowns principle: run lightweight LLM-based embedding and clustering over production traces to surface topic clusters, identify what users are actually asking, detect sentiment, and find failure modes that were not anticipated at build time. Wealth advisors should be included as annotators per the Dual Persona Requirement to validate whether the clustered topics represent real quality issues.

// What mistakes should you avoid when designing agent observability?

Assuming that an existing Grafana or Datadog contract solves the agent observability problem — it handles technical observability only, not functional observability.
Excluding domain experts (clinicians, lawyers, wealth advisors) from the observability workflow because it 'feels too technical' — this is the exact population that can evaluate agent quality and their participation is what separates good teams from average ones.
Collecting human annotation grades without capturing written justifications — the justifications are the mechanism by which you build scalable automated scoring; grades alone do not transfer.
Underestimating the data infrastructure requirements of agent traces — treating them like standard log or metric data will cause ingestion, query performance, and real-time visibility failures at scale.
Ignoring the full-text search requirement across trace content — without text-based indexing, you cannot answer basic operational questions like 'show me every trace where the agent mentioned a specific topic or entity'.
Conflating observability and evals as separate systems requiring separate infrastructure — they are the same problem solved by the same system; the only difference is batch-vs-realtime and known-vs-unknown inputs.
Treating agent observability as purely a technical persona problem — if only engineers are looking at traces, you are missing the qualitative signal that domain experts can provide.

// What key terms should you know for agent observability?

Agent Observability: The practice of monitoring, measuring, and improving the quality and behavior of AI agent systems in production — encompassing both technical metrics and functional/qualitative properties of agent outputs.
Traditional Observability: Established monitoring practice focused exclusively on uptime and technical performance — latency, error counts (400/500 level), duration — using tools like Grafana and Datadog. Answers the question: is the system operational?
Functional Observability: The qualitative layer of agent observability: was the agent's response grounded in retrieved context, did it use the expected tools, was it aligned to the brand standard set in the system prompt? Requires purpose-built agent tooling.
Technical Observability: The metrics-level layer of observability applicable to both traditional and agent systems: latency, time to first token, total tokens, duration, cache hits, error rates. Comes 'on the house' when an agent is properly traced.
Agent Traces Are Nasty: The characterization of agent trace data as highly semi-structured, containing large volumes of unstructured text, potentially exceeding a gigabyte per trace or 20 megabytes per span, requiring specialized database infrastructure to ingest, index, and query in real time.
Non-Deterministic: The property of agent/LLM systems where the reasoning path and output vary across identical or similar inputs due to the high variety and abstraction of language models — contrasted with the deterministic, known control flow of traditional applications.
Known Unknowns: Anticipated failure modes in agent behavior that can be defined in advance and measured with automated scoring functions (e.g., groundedness checks, tool-use verification).
Unknown Unknowns: Emergent patterns, failure modes, and usage behaviors in production agent traces that were not anticipated at build time — surfaced through LLM-based embedding, clustering, and topic modeling over production traces.
Human Annotation: The workflow of having domain experts (not just engineers) review production agent traces, assign quality grades, and write justifications for those grades — which then seed the development of scalable automated scoring functions.
Dual Persona Requirement: The principle that effective agent observability requires both technical personnel and non-technical domain experts (clinicians, lawyers, wealth advisors, etc.) to participate, because domain experts are closest to the users or problem space and can evaluate qualitative agent quality.
Iteration Loop: The cycle between detecting a problem in production observability traces and validating a fix through offline experimentation/evals — the goal of agent observability infrastructure is to make this loop faster and more direct.
Write-Ahead Log: A database mechanism used in agent trace infrastructure to immediately persist incoming trace data so users can see interactions in true real time as they occur.
Tantivy Index: A full-text indexing approach (based on a forked open-source Rust framework similar to Apache Lucene) required for agent observability databases to support text-based search queries across unstructured content within traces.
Trace: A complete record of a full agent interaction or workflow, from start to finish.
Span: A single step within a trace — for example, one model call or one tool call within a larger agent interaction.

// FREQUENTLY ASKED QUESTIONS

What is agent observability?

Agent observability is the practice of monitoring, measuring, and improving the quality and behavior of AI agent systems in production. Unlike traditional observability, it covers both technical metrics (latency, tokens, errors) and functional properties — whether the agent's output was grounded in retrieved context, used the right tools, and met quality standards. It's a broader, qualitative discipline built for non-deterministic LLM systems.

What is the difference between traditional observability and agent observability?

Traditional observability is entirely about uptime and technical performance — is the app up and delivering the expected experience. Agent observability adds a functional layer that measures qualitative, semantic, and reasoning-level properties of agent behavior. Traditional tools like Datadog answer 'is the system operational?' while agent observability answers 'was the agent's output correct, grounded, and high quality?'

How do I know if my system needs agent-specific observability?

Classify your system's determinism profile first. If it follows known control flow with deterministic code paths, traditional observability may suffice. If it uses an LLM where the reasoning path is variable and non-deterministic, you need agent observability. Any system where you must measure why an agent took one path versus another — or evaluate response quality — requires agent-specific tooling layered on top.

How do I set up human annotation for agent traces?

Establish a process where domain experts — clinicians, lawyers, wealth advisors — review production traces, assign quality grades, and provide written justifications for those grades. The justifications are critical: they become the raw material for building scalable, automated scoring functions. Grades alone don't transfer, so never skip the justifications. Find failure modes through human review first, then systematize them.

Can I just use Datadog or Grafana for my AI agent?

No — Datadog and Grafana handle technical observability only (latency, uptime, 400/500 errors), not functional observability. They can't evaluate whether your agent's output was grounded, used the right tools, or met quality standards. Keep these tools for the technical layer, but layer agent-specific observability on top for functional quality. They're complementary, not competitive.

How does agent observability compare to running evals?

Observability and evals are the same problem solved by the same underlying system. The only difference: with evals you know the inputs ahead of time and run them in batch, while with observability you don't know the inputs ahead of time and process them in real time. Treating them as separate systems requiring separate infrastructure is a common, costly mistake.

When should I use the Hetzel observability framework?

Use it whenever you're designing, auditing, or advising on the observability strategy for an AI agent system. Also apply it when stakeholders ask why existing tools like Grafana or Datadog are insufficient for agent workloads. It's most valuable at the point of choosing your observability stack, before you've committed infrastructure to an approach that can't handle agent traces.

Why are agent traces so hard to store and query?

Agent traces are highly semi-structured, contain massive volumes of unstructured text, can exceed a gigabyte per trace with individual spans reaching 20 megabytes, and must still be delivered in real time. Traditional observability databases — even OLAP tools like ClickHouse — were not designed for this. You need full-text indexing and write-ahead log ingestion to handle the volume and query patterns.

What results can I expect from applying this framework?

You'll correctly diagnose which observability layer your system needs, avoid wasting budget on tools that can't measure agent quality, and build a workflow that surfaces both anticipated failure modes (known unknowns) and emergent ones (unknown unknowns). Most importantly, you'll compress the iteration loop between spotting a problem in production and validating a fix in experimentation.

What are known unknowns and unknown unknowns in agent observability?

Known unknowns are failure modes you can anticipate and build automated scores for — like groundedness checks or tool-use verification. Unknown unknowns are patterns and issues you discover by running clustering and topic modeling over production traces. A complete agent observability strategy must address both; automated scores alone miss emergent behaviors you never anticipated at build time.

Who should have access to agent observability data?

Both technical personnel and non-technical domain experts. Traditional observability is consumed exclusively by systems or product engineers. Agent observability, done well, must include domain experts — clinicians, lawyers, wealth advisors — who are closest to the users or problem space and can evaluate qualitative agent quality in natural language. Excluding them means missing the qualitative signal that separates good teams from average ones.

// GET THIS SKILL — FREE