Frequently Asked Questions About Hetzel Agent Observability Differentiation Framework
22 answers covering everything from basics to advanced usage.
// Basics
What exactly is functional observability?
Functional observability is the qualitative layer of agent observability: was the agent's response grounded in retrieved context, did it use the expected tools, was it aligned to the brand standard set in the system prompt, and was the response high quality? Unlike technical metrics, functional observability requires purpose-built agent tooling — it doesn't come automatically from tracing.
What is technical observability and does it come free?
Technical observability covers latency, time to first token, total tokens, duration, cache hits, and error rates — metrics applicable to both traditional and agent systems. It comes 'on the house' when you properly trace an agent, meaning you get it automatically as a byproduct of instrumenting the system. Functional observability is the part you have to build deliberately.
What does 'agent traces are nasty' mean?
It's the characterization of agent trace data as highly semi-structured, containing large volumes of unstructured text, potentially exceeding a gigabyte per trace or 20 megabytes per span, and requiring real-time delivery. This combination of size, structure, and latency makes it a completely new systems problem that traditional observability infrastructure was never designed to handle.
What's the difference between a trace and a span?
A trace is a complete record of a full agent interaction or workflow, from start to finish. A span is a single step within that trace — for example, one model call or one tool call within the larger interaction. A single trace contains many spans, and both can grow large in agent systems.
// How To
How do I classify whether my system is deterministic or non-deterministic?
Ask whether your system follows known control flow with deterministic code paths, or whether it uses an LLM where the reasoning path is variable. Traditional applications have known control flow. Agents are non-deterministic because LLMs have high variety and are abstracted — the same input can produce different reasoning paths. If yours is the latter, proceed to full agent observability.
How do I audit the scope of what needs to be measured?
Separate technical metrics (latency, duration, time to first token, total tokens, error counts, cache hits) from functional metrics (groundedness in retrieved context, correct tool usage, alignment to brand standard, response quality). Traditional tools cover the technical set. Agent observability tooling is required for the functional set. Map each metric you care about to one of these two categories to reveal your tooling gap.
How do I map the read patterns my consumers need?
Identify whether consumers need real-time streaming visibility as interactions happen, SQL/analytical queries over historical traces for experimentation, or full-text search across trace content like 'show me every trace that mentioned word X.' If all three are needed, a purpose-built agent trace database or stack is required — a standard observability backend won't cover all three patterns.
How do I surface unknown unknowns in production traces?
Implement lightweight LLM-based embedding and clustering over your production traces to surface emergent topics, user intent patterns, sentiment signals, and unexpected failure modes you didn't anticipate at build time. This is distinct from known unknowns, where you define explicit automated scores in advance. Clustering and topic modeling reveal what you didn't know to look for.
How do I close the iteration loop between production and experimentation?
Once issues surface in production traces, make it fast to add those traces to an offline dataset, run evals against them in batch, and experiment with agent changes. The goal is compressing the time between 'problem seen in production' and 'fix validated in experimentation.' Because observability and evals are the same system, this loop should be direct, not a handoff between disconnected tools.
// Troubleshooting
My agent trace queries are slow at scale — what went wrong?
You likely underestimated the data infrastructure requirements of agent traces and treated them like standard log or metric data. Agent traces need full-text indexing (such as a Tantivy index) and write-ahead log ingestion for real-time visibility. Traditional observability databases, even OLAP tools like ClickHouse, are insufficient without augmentation for the volume and unstructured text agent traces contain.
I can't answer 'show me every trace that mentioned X' — why?
You're missing full-text search across trace content. Without text-based indexing, you cannot answer basic operational questions about which traces reference a specific topic or entity. This requires a full-text indexing approach like a Tantivy index (based on a forked Rust framework similar to Apache Lucene) built into your agent observability database.
My automated scores work but I keep missing failures — why?
Automated scores only cover known unknowns — the failure modes you anticipated. You're missing unknown unknowns: emergent patterns you didn't build scores for. Run clustering and topic modeling over production traces to surface these. A complete strategy addresses both; relying only on predefined scores guarantees you'll miss behaviors you never thought to measure.
My human annotations aren't producing scalable scoring — what's wrong?
You're probably collecting grades without written justifications. Grades alone don't transfer into automated scoring. The justifications are the mechanism by which you build scalable, automated scoring functions — they explain the reasoning behind each grade, which becomes the seed for systematizing the judgment. Always capture why a reviewer graded a trace the way they did.
// Comparisons
Should I replace Datadog and Grafana with an agent observability platform?
No. Traditional tools like Datadog and Grafana remain valid and useful for the technical layer — 400/500 errors, uptime, website-level performance. Don't replace them; layer agent observability on top for functional quality. The two are complementary, not competitive, at this layer. Trying to replace them wholesale leaves gaps in technical monitoring they already cover well.
How does agent observability compare to a generic monitoring approach?
Generic monitoring measures whether the system is operational — uptime, latency, errors. Agent observability adds a functional layer that measures the semantic and reasoning quality of outputs, includes domain experts as consumers, handles nasty semi-structured trace data, and unifies with evals. A generic approach will tell you the agent responded quickly but nothing about whether the response was correct or grounded.
Is ClickHouse enough for agent trace storage?
Likely not on its own. ClickHouse is an OLAP tool suited to analytical queries, but agent traces are semi-structured, contain large unstructured text, and require full-text search plus real-time streaming. Without augmentation for full-text indexing and write-ahead log ingestion, even ClickHouse is insufficient for the complete set of read patterns agent observability demands.
How is this framework different from just adding LLM eval scores?
Eval scores are one piece — the known unknowns layer. This framework is broader: it diagnoses your system's determinism, separates technical from functional observability, assesses trace data infrastructure, maps stakeholder personas including domain experts, designs human annotation workflows, and closes the production-to-experimentation loop. Eval scores alone don't handle real-time observability, unknown unknowns, or the dual persona requirement.
// Advanced
Why do domain experts matter more than engineers for agent quality?
Domain experts — clinicians, lawyers, wealth advisors — are closest to the users or problem space and can evaluate qualitative agent quality in natural language. Engineers can measure latency and errors, but they can't judge whether a clinical recommendation was appropriate or whether financial advice was sound. The dual persona requirement exists because functional quality is a domain judgment, not a technical one.
How does the write-ahead log enable real-time visibility?
A write-ahead log immediately persists incoming trace data so users can see interactions in true real time as they occur, rather than waiting for batch processing. In agent observability, where traces can be huge and consumers need live visibility into what agents are doing, write-ahead log ingestion is a key mechanism for delivering the real-time read pattern.
How do human justifications turn into automated scoring functions?
When domain experts grade traces and explain their reasoning, those justifications reveal the specific criteria that distinguish good from bad outputs. You systematize these criteria into automated scoring functions — for example, an LLM-based scorer that checks the same properties the human described. Human review finds the failure modes; the written justifications provide the template for scaling that judgment automatically.
What's the relationship between clustering and finding new failure modes?
Clustering and topic modeling over production traces group similar interactions together, surfacing emergent patterns — user intents, sentiment signals, and failure modes you never anticipated. Once a cluster reveals a recurring problem, you can promote it from an unknown unknown to a known unknown by defining an automated score for it. It's the discovery engine that feeds your automated scoring pipeline.
Can one system really solve both observability and evals?
Yes — observability and evals are the same problem solved by the same underlying system. Evals run known inputs in batch; observability processes unknown inputs in real time. Building separate infrastructure for each duplicates effort and fragments your iteration loop. A unified system lets you take a problematic production trace directly into a batch eval, compressing the fix-validation cycle.