Where would the answer come from?
In one line
Every answer a model gives has one of three sources — its training, your prompt, or a tool — and asking which one, before building, decides most of the architecture.
The situation
A colleague demonstrates the new assistant: "What's our refund window for annual plans?" It answers, instantly and warmly, "Annual plans can be refunded within 30 days of purchase." Everyone nods.
The refund window is 14 days. It was 30 days two years ago, and it is 30 days at three competitors whose help pages the model read during training. Nothing in the demo distinguished a remembered fact from a looked-up one, and nobody asked — because the question that would have caught it is not one people think to ask of software.
The three sources#
A model can only produce an answer from one of three places, and each has a different failure mode.
Training data. Everything the model absorbed during : broad, frozen at a cut-off date, drawn from the public internet, uncitable. Ask it what a mutex is and this source is excellent. Ask it your refund window and it will produce the industry's most common answer, fluently, because that is what "most likely continuation" means.
The prompt. Whatever you put in front of it on this call: instructions, examples, and — critically — documents you fetched for the purpose. This is , and it is the source that can be current, private, and cited. Its failure mode is fetching the wrong document, or none, and getting a confident answer anyway.
A tool. The model asks your code to run something — look up an order, query a database — and the result comes back exact. This is . Its failure mode is the model calling the wrong tool, or your code trusting the call without checking who asked.
Sorting the demo#
Run the demo question through that list. "What is our refund window" is a fact about your business, so training data is the wrong source by construction. It changes, so it belongs in retrieval — the policy page, fetched at answer time, with a citation the user can click. If the question had been "what is the refund window for my order", that is a tool: it depends on this customer's plan and purchase date, and no document contains the answer.
Once each question has a source, the architecture largely writes itself, and so does the test. Retrieval questions are tested by checking the right document was fetched. Tool questions are tested by checking the right call was made. And questions that can only come from training data get a design decision: is a plausible-sounding answer acceptable here, or should the assistant decline?
The demo's answer was a in the precise sense — an answer with no support in anything the model was given — and it was invisible because the source was never stated. "Show me where that came from" is the question that would have caught it, and it is worth making the assistant answer it every time.
Questions to ask
- Is this a general fact, a fact about us, or a fact about this user?
- Does the answer change? If yes, it cannot live in training data.
- Could the assistant show me its source? If not, why would I trust it?
- What should happen when the source has nothing to say?
Mindset
Before asking whether a model can answer something, ask where the answer would come from. Training data for the general, retrieval for the specific and changing, tools for the exact and personal. The question takes ten seconds and it is the difference between a demo and a system.
Goes deeper in
Phase 2 · 6 min
Training vs inference
Training is a write path you will almost certainly never run; inference is a stateless read path, and conflating the two is the source of most bad architecture decisions here.
Phase 3 · 7 min
Retrieval & RAG
Fetch the relevant text at request time and put it in the prompt, because almost everything worth answering about is either private, recent, or needs a citation.
Phase 4 · 7 min
Agents & tool use
A loop where the model picks a tool and your code decides whether to run it, and every interesting problem is in the loop rather than in the model.
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.