What a coding assistant should see on every keystroke
In one line
Every keystroke is a model call with a fixed budget; what goes in the window, in what order, decides both what the assistant can know and what each call costs.
The situation
An internal coding assistant suggests completions as engineers type. It is good when the answer is in the current file and useless when it is not — it does not know the team's conventions, cannot see the function being called from another file, and suggests a logging library the codebase does not use. Meanwhile the per-call cost is high, because the prompt is rebuilt from scratch on every keystroke.
Both complaints are the same problem: nobody decided what belongs in the , so the window contains whatever was easiest to include.
Write the budget down#
Say the window budget per call is 8,000 tokens. Allocate it:
system prompt, task rules 600 fixed
repo conventions, style, stack 900 fixed per repo
the open file, around the cursor 2,500 changes slowly
related files (imports, callers) 2,000 changes on navigation
recent edits and the cursor line 500 changes every keystroke
headroom for the completion 1,500
Now every complaint maps to a line. "Doesn't know our conventions" is the second line, which was missing. "Can't see the function I'm calling" is the fourth, which needs a cheap step — pull in the definitions of the symbols near the cursor. "Suggests the wrong library" is fixed by the conventions line too, which is where the list of approved dependencies lives.
The budget also says what to leave out. The whole file is not needed, only a window around the cursor. The entire git history is not needed. Old completions are not needed. Everything that goes in displaces something else.
Order by how often it changes#
The and repo conventions are identical across thousands of calls. The open file changes every few seconds. The cursor line changes every keystroke. Put them in that order.
reuses the computed prefix when the start of a prompt matches a previous one exactly. With stable content first, the first 1,500 tokens are prefilled once and served from cache thereafter; only the volatile tail is fresh. That is the difference between paying for 8,000 tokens per keystroke and paying for roughly 3,000 — and it is free, because it is only an ordering.
Put the cursor line at the very end, immediately before the completion. The model attends most reliably to the end of the prompt, which is where the thing it is completing should be.
Questions to ask
- Is the budget written down, line by line, or is the window just "whatever fits"?
- Which line does each quality complaint map to?
- What is stable across calls, and is it at the front?
- What is in the window that the assistant never uses?
Mindset
The context window is a stack frame you assemble by hand on every call. Write the budget, put the stable parts first, and put the thing being worked on last. Most quality and most cost complaints turn out to be allocation problems.
Where it connects
Phase 2 · 7 min
Prefill & decode
One request is two different workloads with two different bottlenecks, and every latency number you care about belongs to one or the other.
Phase 3 · 6 min
Context engineering
Deciding what occupies a finite window, in what order, on every call — and it is mostly a discipline of leaving things out.
Got the shape of it?
Examples do not count toward phase progress — that stays on the topics. This is just so the list remembers what you have seen.