The bot quotes last year's policy, with a citation
In one line
Retrieval worked exactly as built — it found the most similar document, which was the stale one — and the fix is in what gets indexed and logged, not in the prompt.
The situation
A customer asks the assistant about the cancellation window. It answers "30 days", cites cancellation-policy.md, and is wrong: the window changed to 14 days in January. The engineer checks the prompt, which clearly says to answer from the provided documents. The assistant did exactly that. The document it was provided was the old one.
This is the single most common failure in production, and it is worth understanding as a retrieval failure rather than a model one — because every instinct points at the model.
What actually happened#
When the policy changed, the new page was added to the knowledge base. The old page was never removed — it was archived, renamed, or left in a folder the ingest job still reads. Both versions were chunked and embedded. At query time the step returned the five most similar chunks, and the old version, being slightly closer in wording to the question, edged out the new one. The model was handed a stale passage and did its job faithfully.
Three things make this hard to catch. It is silent: no error, no empty result, a normal-looking citation. It is intermittent: on a differently phrased question the new chunk wins, so the bug does not reproduce reliably. And it looks like a , so the investigation starts at the wrong end of the pipeline.
The fixes are all upstream of the model#
Log what was retrieved. Every request should record the chunk ids and scores that went into the prompt. With that one field, this bug takes five minutes to diagnose instead of a week. Without it, every retrieval failure looks like a model failure.
Make the index reflect the truth. Ingest should delete or supersede, not only add. Each chunk carries metadata — source, version, effective date — and retrieval filters on it: current documents only, or the latest version per source. Superseded content should not be retrievable at all for a "what is the policy" question.
Measure on a schedule. Take a hundred real questions with known correct sources and check, nightly, that the right chunk is in the top five. This catches the next stale document before a customer does, and it catches the slower version of the same problem — an index that degrades as the corpus grows.
Show the source's date. If the citation says "cancellation-policy.md, last updated March 2024", a human reader has a chance of noticing. Surfacing provenance costs nothing and catches what the pipeline misses.
Questions to ask
- Can I see, for any given answer, exactly which chunks were in the prompt?
- When a document changes, what happens to the old chunks?
- Does retrieval filter on version or date, or only on similarity?
- Is retrieval recall measured, or only end-to-end answer quality?
Mindset
When a RAG system answers wrongly, check what was retrieved before touching anything else. The model can only be as good as the passages it was handed — and a stale passage, faithfully used, produces a confident, cited, wrong answer.
Where it connects
Phase 3 · 7 min
Retrieval & RAG
Fetch the relevant text at request time and put it in the prompt, because almost everything worth answering about is either private, recent, or needs a citation.
Phase 3 · 6 min
Vector indexes
An index that is allowed to be wrong, where the tuning dial is recall and correctness is something you measure rather than assume.
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.