Retrieval & RAG

The generation half of RAG is a prompt. The retrieval half is an information retrieval system, and that is where essentially all the engineering lives:

1. Chunking — split documents into retrievable passages
2. Embedding — turn each chunk into a vector, store it with its metadata
3. Query — embed the question, find nearest neighbours
4. Reranking — rescore the shortlist with a stronger model
5. Assembly — put the survivors in the prompt with instructions

The most useful diagnostic habit in this whole area follows from that list. When a RAG system gives a bad answer, check what was retrieved before touching the prompt. Most of the time the right passage was never in the context, and no amount of prompt engineering can make a model use text it did not receive.

That is also why Recall is the metric that matters. You can have a perfect generator and a useless system if the search misses one relevant document in three.

Chunk boundaries determine what can be retrieved, and they are chosen before any query exists. Get them wrong and no downstream component recovers.

Too small and chunks lose their context. A paragraph reading "This does not apply to enterprise accounts" is meaningless without knowing what "this" was, and it will be retrieved for queries it actively misleads.

Too large and precision collapses. A 4,000-token chunk retrieved for one relevant sentence drags 3,900 tokens of noise into the prompt, crowding out other results and diluting attention.

The starting point that works for most prose is 400 to 800 tokens with chunk overlap of 10 to 15 per cent, so an idea split across a boundary still appears whole somewhere. Better than any fixed number, though, is splitting on structure: a heading, a section, a function, a table. Documents already contain the author's own boundaries, and those are usually better than anything a character counter will find.

Two refinements repay their cost quickly. Contextual prefixing prepends a line of document and section context to each chunk before embedding, so a chunk carries where it came from — this is one of the largest single quality gains available and costs a few extra tokens per chunk. And metadata on every chunk — source, date, section, permissions — lets you filter before searching, which is both a relevance win and, for permissions, a correctness requirement.

The reflex when RAG underperforms is to reach for a bigger generator. The ordered list of things that actually help looks nothing like that.

Add keyword search. Hybrid search runs BM25 alongside the vector query and merges the rankings. Embeddings are poor at exact identifiers — error codes, SKUs, surnames, version numbers — and keyword search is perfect at them. One extra query, and it removes most of the embarrassing misses.

Add a reranker. A Cross-encoder scoring the top fifty candidates sees query and document together, which no pair of independently computed Embeddings can. It typically improves what reaches the prompt more than any embedding upgrade, and it only runs on a shortlist so the cost is bounded.

Rewrite the query. "What about the other one?" is unanswerable standalone. Rewriting it against conversation history into "What is the refund window for annual plans?" fixes a large class of multi-turn failures for one cheap model call.

Then, maybe, a better embedding model. It is the expensive option — a full corpus re-embed and index rebuild — and usually the smallest gain of the four.

Putting sources in the prompt reduces Hallucination substantially. It does not eliminate it, and the residual failure modes are specific enough to design against.

The model can contradict the passages, blending retrieved text with training knowledge. It can answer from parameters when retrieval returned nothing useful, which is the most dangerous case because the answer looks identical. And it can cite a passage that does not support the claim it is attached to.

Three mitigations, in order of cost. Instruct explicitly that the answer must come from the provided context and that "I don't know" is an acceptable response — cheap, and it works more often than it sounds like it should. Require inline citations by chunk id, then verify programmatically that every cited id was actually retrieved; a fabricated id is a bug you can catch in code. And for high-stakes paths, run a second cheap call that checks whether each claim is supported by the cited passage.

Note that this is where RAG and Fine-tuning divide cleanly. Retrieval carries facts, because facts change. Fine-tuning carries form, because form does not.

An internal documentation assistant over 2,000 pages. First version: 1,000-token fixed chunks, vector search, top 3, straight into the prompt. Accuracy on 100 real questions: 61%.

Failures, categorised by hand — which is the step people skip:

retrieval missed the answer entirely      27 cases
right document, wrong chunk               8 cases
model ignored the context                 4 cases

Thirty-five of thirty-nine failures were search. Changes, measured one at a time:

baseline                                            61%
+ split on headings instead of fixed size           68%   (+7)
+ prepend doc title and section to each chunk       74%   (+6)
+ hybrid search with BM25                           79%   (+5)
+ cross-encoder rerank, top 50 -> top 5             86%   (+7)
+ query rewriting against chat history              89%   (+3)

61% to 89%, and the generator never changed. The two interventions that look least like AI work — splitting on headings and adding keyword search — accounted for twelve points between them.