It got worse and nothing changed
In one line
Every dashboard you already had was green; the one that moved is one you would only have if you built it for this kind of system.
The situation
Resolution rate on the support assistant drifts from 79% to 68% over four days. No deploys. No errors. Latency flat. Model version pinned. The on-call engineer opens every dashboard the team has and every one is green, and the only evidence anything is wrong is a Slack thread of customers saying the answers have got worse.
The failure lives between the dashboards#
A request through this system is a chain: rewrite the query, retrieve passages, rerank, call the model, validate, sometimes call a tool. The dashboards measure the chain's endpoints — request in, response out, how long, did it error. Everything in between is invisible, and everything in between is where this class of failure lives.
The model returned fluent, well-formed, confident answers throughout. They were worse because the passages it was handed were worse. Retrieval — the fraction of questions where the right chunk reached the prompt — had fallen from 0.91 to 0.62, and there was no line on any graph for it.
Instrument the chain, not the call#
A per request, with one per stage, is the difference between a four-day investigation and a five-minute one. Each span carries what you would need to reconstruct the decision: the retrieved chunk ids and scores, the reranked survivors, token counts, cache hits, the raw output before parsing.
With that in place, the diagnosis is a query: retrieval scores by day. Top-1 score dropped on the day a documentation import added forty thousand auto-generated chunks that now dominate every result. A metadata filter and a reindex; half an hour.
Then add the this kind of system needs and ordinary services do not. Retrieval recall, sampled nightly against a labelled set. Refusal and fallback rate, which moves first when a provider ships a new checkpoint — the case, where your code did not change and the model did. Output-length p99, the tail-latency driver. Tokens in and out per stage, which is the cost curve before the invoice. All four move days before a customer writes in.
And close the loop: the requests that went wrong this week become eval cases next week. Traces that never become tests are logs nobody reads.
Questions to ask
- For any given answer, can I see which passages were in the prompt?
- Is retrieval quality a metric, or only end-to-end quality?
- What moves first when the provider updates the model behind the same name?
- Which of this week's bad traces became test cases?
Mindset
The characteristic failure here is a 200 with the wrong content. Trace every stage, log what was retrieved, and watch the leading indicators — because the dashboards built for services that fail loudly will stay green through the entire incident.
Where it connects
Phase 3 · 7 min
Retrieval & RAG
Fetch the relevant text at request time and put it in the prompt, because almost everything worth answering about is either private, recent, or needs a citation.
Phase 4 · 6 min
Observability
Aggregate metrics tell you something regressed; only a trace of the whole chain tells you which of six stages did it.
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.