How to Understand How Users Use Your Wealth Agent
For Fintech and wealth management agent builders · Based on Hetzel Agent Observability Differentiation Framework
// TL;DR
Fintech teams building wealth management agents often don't yet know what questions to ask about production usage. Traditional observability shows uptime and error rates but never reveals user intent, sentiment, or emergent patterns. The Hetzel framework treats this as an unknown unknowns problem: run LLM-based embedding and clustering over production traces to surface what users actually ask, detect sentiment, and find unanticipated failure modes. Wealth advisors are brought in as annotators to validate whether clustered topics represent real quality issues. Use this when you need discovery, not just dashboards.
Why can't my error dashboards tell me how users actually use the agent?
Your wealth management agent is live and your dashboards are green — but you have no idea what users are really asking or whether the answers are any good. That's expected. Traditional observability will show you uptime and error rates, but it will not reveal user intent, sentiment, or emergent usage patterns.
This is fundamentally an unknown unknowns problem. You don't know what questions to ask yet, so predefined metrics can't help. You need a discovery mechanism, not another latency chart.
How do I discover what users are actually asking?
Apply the unknown unknowns branch of the Known Unknowns vs. Unknown Unknowns principle. Run lightweight LLM-based embedding and clustering over your production traces. This surfaces topic clusters — the real categories of questions users bring — detects sentiment, and finds failure modes nobody anticipated at build time.
Maybe you discover a large cluster of users asking about tax-loss harvesting the agent handles poorly, or a sentiment dip whenever the agent hedges on specific fund recommendations. None of that shows up in error rates. Clustering is how you convert a firehose of unstructured traces into a map of actual usage.
Who should validate whether a cluster is a real problem?
Wealth advisors. Under the Dual Persona Requirement, domain experts closest to the clients should be included as annotators to validate whether the clustered topics represent genuine quality issues. A cluster of questions about a particular product isn't automatically a problem — an advisor can tell you whether the agent's responses in that cluster were sound or subtly misleading.
When advisors grade traces, capture their written justifications through human annotation. Those justifications become the seed for automated scoring functions — for instance, a scorer that flags responses giving unsuitable advice for a client's stated risk profile. This is how you promote a discovered issue from an unknown unknown into a monitored known unknown.
What technical realities should I plan for?
Wealth agent traces carry full conversation context and often retrieved market data or client account details, so they're semi-structured and can be large — the Agent Traces Are Nasty pattern. Plan for full-text indexing so you can answer questions like 'show me every trace that mentioned a specific ticker or product,' and write-ahead log ingestion so compliance and product teams get real-time visibility.
And don't rip out your existing tools. Datadog or Grafana still handle the technical layer — errors, uptime, performance. Layer agent observability on top for functional quality; the two are complementary.
What's my first move?
Stand up a clustering pass over your last few weeks of production traces before you write a single new metric. Let the data show you the real usage map, then sit two wealth advisors down to grade the most surprising clusters with justifications. Within one cycle you'll know which emergent behaviors deserve automated scores — and you'll have turned 'we don't know what to ask' into a prioritized quality backlog.
// FREQUENTLY ASKED QUESTIONS
How do I find out what users ask my wealth agent when I don't know what to look for?
Run LLM-based embedding and clustering over your production traces. This surfaces topic clusters revealing the real categories of user questions, detects sentiment, and finds unanticipated failure modes. It's the unknown unknowns branch of the framework — a discovery mechanism that works precisely when you don't yet know which metrics to define.
Why should wealth advisors review agent traces?
Per the Dual Persona Requirement, wealth advisors are closest to clients and can validate whether clustered topics represent real quality issues. A cluster isn't automatically a problem — an advisor judges whether the agent's responses were sound or subtly misleading. Their written justifications also seed automated scoring functions for things like suitability of advice.
Should I stop using Grafana for my fintech agent?
No. Grafana and Datadog remain valid for the technical layer — errors, uptime, and performance. Layer agent observability on top for functional quality like advice suitability and groundedness. The two are complementary, not competitive. Removing your traditional tools would leave gaps in technical monitoring they already handle well.