The bot forgot my name
In one line
Inference is stateless; what feels like memory is the client re-sending the transcript, and the moment the transcript is trimmed to fit, the earliest facts are the first to go.
The situation
A customer introduces themselves — "I'm Priya, order 4471" — and twenty minutes into a long conversation the assistant asks for their order number. Support flags it as "the bot has no memory". Half right: it never had any. The interesting part is why it seemed to for the first nineteen minutes.
There is no session on the server#
Every model call is over frozen weights: input in, output out, nothing written. The model at turn 20 has no more knowledge of turn 1 than a pure function has of its previous invocation. What makes a conversation feel continuous is that the client sends the entire transcript on every turn. Turn 20's prompt contains turns 1 through 19, so the model can "remember" Priya — by re-reading her introduction each time.
Two consequences follow immediately. Each turn costs more than the last, because the prompt grows by the whole conversation so far. And the is finite, so at some point the transcript must be trimmed, and the natural place to trim is the beginning. Which is where Priya introduced herself.
Nothing broke. The system did exactly what it was built to do, and the design had no policy for what to keep.
Deciding what survives#
Memory, in this setting, is a design decision about the prompt, and there are three moves worth knowing.
Keep a running summary. Every few turns, compress the older transcript into a short paragraph of established facts — name, order, issue, what has been tried — and send that instead of the raw turns. The summary is small, stable, and sits early in the prompt where can reuse it.
Extract structured state. If the conversation establishes a customer name and an order id, put them in a small object your code maintains and injects at the top of every prompt. That is not "memory" in any model sense; it is a session variable, and it survives any amount of trimming because it is never trimmed.
For anything that should persist between conversations — preferences, past issues — store it and fetch it. That is over the user's own history, and it is the only mechanism that makes "remember me next week" true.
The mistake to avoid is the one the first version made: treating the raw transcript as the state, and letting the truncation policy decide what the assistant knows.
Questions to ask
- What facts, once established, must never fall out of the window?
- What is the truncation policy, and who decided it?
- Is there state that belongs in code rather than in the prompt?
- What should carry across conversations, and where is it stored?
Mindset
The model has no memory; the prompt is the memory, and you assemble it every turn. Decide explicitly what survives, or the context limit will decide for you, starting with the oldest thing the user said.
Where it connects
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.