How to Build Infrastructure for Nasty Agent Traces

Why do agent traces break my existing observability database?

Because agent traces are nasty. They're highly semi-structured, contain massive volumes of unstructured text, can exceed a gigabyte per trace with individual spans reaching 20 megabytes — and they must still be delivered in true real time. This is a completely new systems problem that traditional observability infrastructure was never designed to handle.

If you treat these like standard log or metric data, you'll hit ingestion bottlenecks, query performance collapse, and real-time visibility failures at scale. The Agent Traces Are Nasty principle exists to make you plan for this up front instead of discovering it in production.

What read patterns do I actually need to support?

Map the required read patterns before choosing a backend. Consumers typically need three things: real-time streaming visibility as interactions happen, SQL/analytical queries over historical traces for experimentation, and full-text search across trace content — for example, 'show me every trace that mentioned the word X.'

If all three are needed — and for a serious agent platform they usually are — a purpose-built agent trace database or stack is required, not a standard observability backend. Each pattern stresses your storage layer differently: streaming needs low-latency writes, analytics needs columnar scans, full-text needs an inverted index.

Is ClickHouse enough on its own?

Probably not. Even OLAP tools like ClickHouse are likely insufficient without augmentation for full-text indexing and write-ahead log ingestion. You'll want a write-ahead log so incoming traces persist immediately and users see interactions in true real time. And you'll want a full-text indexing approach — something like a Tantivy index, based on a forked Rust framework similar to Apache Lucene — to serve text search across unstructured trace content.

Without that full-text layer, you literally cannot answer basic operational questions about which traces reference a specific entity or topic. That's a hard requirement, not a nice-to-have.

Should I build separate systems for observability and evals?

No — and this is where architects most often over-engineer. Observability and evals are the same problem solved by the same underlying system. The only difference: with evals you know the inputs ahead of time and run them in batch; with observability you don't know the inputs and process them in real time.

Design one system that handles both. This makes the iteration loop direct: when a problem surfaces in production, you add those traces to an offline dataset, run evals in batch, and experiment with fixes — all within the same infrastructure. Building two disconnected systems duplicates effort and fragments your feedback loop.

Where does traditional infrastructure still fit?

Don't decommission your existing stack. Traditional tools like Datadog or Grafana remain valid for the technical layer — 400/500 errors, uptime, website-level performance. Your agent trace infrastructure layers on top for functional quality and the nasty trace data. They're complementary at this layer, so integrate rather than replace.

What should I architect first?

Start by profiling expected trace size and structure from step three of the framework, then prototype ingestion with a write-ahead log and a full-text index against a representative sample of real agent traces. Validate all three read patterns — streaming, analytical, full-text — before committing. If a candidate backend can't serve all three on realistic trace sizes, it's the wrong foundation, and finding that out in a prototype is far cheaper than in production.