Phase 3 of 4
Building with it
How do I compose a system around a non-deterministic component?
The model is one call in a larger system. This phase is the engineering around it: what you put in the context, what you get back, and how you know it works.
Topics in this phase
Retrieval & RAG
7 minUnreadFetch the relevant text at request time and put it in the prompt, because almost everything worth answering about is either private, recent, or needs a citation.
After Embeddings, Vector space intuition
Vector indexes
6 minUnreadAn index that is allowed to be wrong, where the tuning dial is recall and correctness is something you measure rather than assume.
After Vector space intuition, Retrieval & RAG
Context engineering
6 minUnreadDeciding what occupies a finite window, in what order, on every call — and it is mostly a discipline of leaving things out.
After Prefill & decode, Retrieval & RAG
Structured output
6 minUnreadGetting a schema-valid object back instead of prose, so the boundary between the model and your code stops being a parsing problem.
After Context engineering
Evals
6 minUnreadAutomated tests for a component whose output is not stable, where the assertion is a threshold on a score and the suite is the only thing making changes measurable.
After Structured output
The adaptation ladder
6 minUnreadFive ways to make a general model fit your task, ordered by cost and by how hard each is to undo — and the discipline is to stop climbing the moment the evals stop improving.
After Training vs inference, Evals
Examples that open after this phase
Problem statements that make sense once these topics are done. They unlock at the same 80% mark the next phase does.
The bot quotes last year's policy, with a citation
Retrieval worked exactly as built — it found the most similar document, which was the stale one — and the fix is in what gets indexed and logged, not in the prompt.
after phase 3 · 4 min · #retrieval #debugging #observability
Chunking a contract so the answers survive
A 500-token window cut clause 7.2 in half; the document already had boundaries, and using the author's beats any character count.
after phase 3 · 4 min · #retrieval #search #architecture
Turning invoices into JSON, and the field that got invented
Schema enforcement made every output parse and quietly created a bug — a required field the invoice did not have, which the model filled in anyway.
after phase 3 · 4 min · #extraction #evals #risk
The copy edit that broke refusals
A one-line prompt change moved the overall score by two points and destroyed a capability worth thirty-six — visible only because the eval was split by kind of question.
after phase 3 · 4 min · #evals #prompts #risk
Make it sound like us
One request, two different problems — the facts change weekly and belong in retrieval, the voice is stable and can be taught — and the ladder tells you how far to climb for each.
after phase 3 · 4 min · #prompts #retrieval #product
What a coding assistant should see on every keystroke
Every keystroke is a model call with a fixed budget; what goes in the window, in what order, decides both what the assistant can know and what each call costs.
after phase 3 · 4 min · #context #cost #prompts