Observability

A request is a pipeline, and each stage can fail independently. Recording only the model call throws away most of the diagnostic surface.

A useful trace captures one Span per stage:

request
├── rewrite query          12 ms
├── retrieve               84 ms   → 20 chunk ids + scores
├── rerank                140 ms   → 5 survivors
├── model call          2,310 ms   → 4,200 in / 380 out tokens
├── validate output         3 ms   → schema ok, 2 citations
└── tool: get_invoice      68 ms   → 200

Attach to each span what you would need to reconstruct the decision: the actual prompt sent, retrieved chunk ids with scores, the raw response before parsing, token counts, model version, and whether a prompt cache hit occurred.

Storing full prompts has a privacy cost, so make it deliberate: redact known PII fields, sample rather than storing everything, and keep a shorter retention window on span payloads than on metrics. But storing nothing makes debugging impossible, and the middle ground — hashes plus a sampled subset — is usually the right call.

For an agent loop, nest spans per iteration. This is what lets you answer "which step of run 4471 went wrong" and "what does step count look like across all runs", which are the two questions that actually come up.

Alongside latency and error rate, four observability signals lead everything else here:

Token counts in and out, per stage. This is your cost curve, visible before the invoice. A prompt that grew by 800 tokens because someone added an example is invisible in every other metric.

Retrieval Recall at k. Sampled against a labelled set, run on a schedule. When end-to-end quality drops, this is the first thing to check, because retrieval is the most common root cause and the least likely to be suspected.

Refusal and fallback rate. The fraction of requests where the model declined, returned an empty answer or hit a validation failure. It is the fastest-moving indicator of a provider model update, and it moves days before anyone files a complaint.

Output length distribution. The p99 of output tokens is your tail latency driver and a good early warning that the model has become chattier after a version change.

Add cost per request as a first-class metric, broken down by stage and route. It is the number that will eventually be asked about in a meeting, and reconstructing it after the fact is painful.

Instrumentation that does not feed back into the Eval suite is just expensive logging.

Sample traces into the golden set. A weekly pull of fifty real requests — especially ones with low retrieval scores, validation failures or unusual lengths — keeps the eval set representative as traffic changes. This is the single habit that makes evals compound.

Every production failure becomes a case. The same rule as before, and tracing is what makes it cheap: you have the input, the retrieved context and the output already.

Run evals on a schedule, not only on deploy. Drift means behaviour changes while your code does not. A nightly run against a pinned suite is how you learn that the provider shipped something.

Capture user signals. A thumbs-down, a copy-to-clipboard, a rephrase of the same question thirty seconds later, an escalation to a human — these are cheap to log and correlate well with quality. A rephrase rate is one of the better proxies for "the answer was not useful" available without asking anyone anything.

A support assistant. Resolution rate drops from 79% to 68% over four days; no deploys, no errors, latency flat.

Without tracing, the investigation is guesswork. With it:

retrieval recall@5    0.91 -> 0.62     over the same four days
refusal rate          4%   -> 5%       flat
output length p99     410  -> 430      flat
model version         pinned, unchanged

Retrieval collapsed. Drill into the spans:

avg chunks returned    5.0 -> 5.0
avg top-1 score        0.81 -> 0.44
chunks from source     "help-centre-v2"  96% of results

A documentation import three days earlier had added 40,000 low-quality auto-generated chunks that were dominating results. The fix was a metadata filter and a reindex — half an hour, once the cause was known.

The point is the shape of the diagnosis. Every aggregate metric except one looked normal. The Hallucinations users were reporting were a symptom two stages downstream of the actual fault, and no amount of prompt work would have touched it.