Attention

In one line

A soft lookup where every token asks every earlier token how relevant it is, and the answer is a set of weights the model learned rather than a rule anyone wrote.

Why it exists

Meaning depends on context in ways no fixed-size window captures. In "the cache that it warms", resolving "it" requires reaching back four tokens. In a 2,000-token document, the referent might be 1,500 tokens back. Any architecture with a fixed reach is wrong for some sentence.

solves this by making the reach learned and dynamic. Each position computes, on the fly, how much every other position should contribute to it. That is the whole mechanism, and it is also where the limit and the cost curve of long prompts both come from.

Resolving "it" is a weighted average over earlier tokens. Nobody wrote the rule that "cache" should win; the weights came out of training.

Query, key and value#

Every token is projected into three vectors by three learned matrices, and the naming is genuinely helpful because it maps onto a lookup.

The query is what this position is looking for. The key is what a position advertises about itself. The value is what actually gets mixed in when a position is selected.

Relevance between position i and position j is the of query i with key j — one number, larger when they point the same way. Compute that for every pair, scale it, run across the row so the weights sum to one, and take the weighted sum of value vectors. That result is added back into position i's representation.

The reason to think of it as a lookup is that a hash table does exactly this with a hard match: one key wins, everything else contributes nothing. Attention softens the match into a distribution, so instead of retrieving one entry you retrieve a blend, weighted by relevance. Everything else about follows from that one substitution.

Heads, and why there are many#

A single attention computation produces one blend per position, which forces the model to express all relationships through one set of weights. Resolving a pronoun and tracking subject-verb agreement would compete for the same distribution.

Instead, each runs many in parallel, each on its own slice of the vector. A 4,096-dimensional model with 32 heads gives each head 128 dimensions to work in. Each head computes its own queries, keys, values and weights, and the outputs are concatenated and projected back to full width.

Heads specialise, and the specialisations are legible enough that interpretability work names them: heads that attend to the immediately previous token, heads that match syntactic structure, heads that copy a rare token mentioned earlier in the prompt. You never assign these roles. They fall out of training the same way a hash function's bit distribution falls out of its construction.

The practical significance is in memory. Every head needs its keys and values kept for the whole sequence, which is what a stores. Architectures that share keys and values across groups of heads — grouped-query attention is the common one — exist precisely to shrink that cache, and they are why modern models can serve long contexts at reasonable batch sizes.

Why long context is expensive#

Attention compares every position with every other position. For a sequence of n tokens, that is n² scores per head per layer. Double the prompt and the attention work quadruples.

This is the origin of the as a hard limit rather than a soft budget. It is also why the cost of a long prompt is not linear in its length, and why "just put the whole document in the context" is a more expensive suggestion than it sounds.

Two things soften it in practice. First, the quadratic term applies to attention only; the networks, which hold most of the parameters, are linear in sequence length. At typical lengths the linear term still dominates total compute, and the quadratic term takes over somewhere in the tens of thousands of tokens. Second, implementations like FlashAttention never materialise the full n² matrix, computing it in tiles that stay in fast on-chip memory. That changes the memory cost from quadratic to linear and gives a large constant-factor speedup, but it does not change the arithmetic count.

In a , the halves the work — position i only attends to positions up to i — which is a constant factor, not a change in the growth rate.

The math

Scaled dot-product attention, in one line:

\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V

Work one position by hand. Suppose a query vector $q = [1, 0, 1]$ and three keys, with $d_k = 3$ so the scale factor is $\sqrt{3} \approx 1.73$ :

k1 = [1, 0, 1]   ->  q·k1 = 2   ->  2/1.73 = 1.16
k2 = [0, 1, 0]   ->  q·k2 = 0   ->  0/1.73 = 0.00
k3 = [1, 1, 0]   ->  q·k3 = 1   ->  1/1.73 = 0.58

Softmax over [1.16, 0.00, 0.58]:

exp:      3.19,  1.00,  1.79        sum = 5.98
weights:  0.53,  0.17,  0.30

The output is 0.53·v1 + 0.17·v2 + 0.30·v3. The first position dominates because its key pointed the same way as the query, but the others still contribute — that softness is what makes the mechanism differentiable, and therefore trainable.

The $\sqrt{d_k}$ divisor is not decoration. Dot products of high-dimensional vectors grow with dimension, and without the scaling, softmax saturates into a one-hot distribution with vanishing gradients.

Worked example

Trace "The cache that it warms" and ask what "it" refers to. At the position of "it", the query is a learned projection of that token's current — which, by this layer, already carries information mixed in from earlier positions.

Scores against the keys of the four preceding tokens, after softmax:

"The"      0.04
"cache"    0.71
"that"     0.05
"it"       0.12    (attending to itself is normal and useful)
"warms"       —    (masked: it comes later)

The output at "it" is now 71% the value vector of "cache". For everything downstream in this layer and the next, "it" carries the content of "cache".

Nothing in that computation knows what a pronoun is. The query and key matrices were shaped by gradient descent to produce large dot products in configurations that reduced prediction error, and this is one such configuration. That is why the mechanism generalises to relationships nobody thought to name — and why it occasionally attends confidently to the wrong thing with no way to inspect the rule, because there is no rule.

Gotchas

Reading attention weights as an explanation. They show where information flowed, not why an answer was produced. There are dozens of layers and hundreds of heads, information moves through the residual stream as well, and published work has repeatedly shown attention maps can be altered without changing the output. Treat them as a debugging hint, never as an audit trail.
Assuming a bigger context window means better use of it. Attention is uniform in capability across the window but not in practice: models reliably weight the beginning and end of a long prompt more heavily than the middle. Put instructions first and the critical passage last, and verify with an eval rather than trusting the advertised limit.
Budgeting long prompts linearly. Between the quadratic attention term and the per-request KV cache, tripling your prompt does not triple your cost — it raises it more than that and cuts how many requests fit on the box at the same time. Measure at the length you actually intend to run.
Expecting attention to retrieve. It is a soft blend over what is already in the context window, not a search over a corpus. If the fact is not in the prompt, no amount of attention will find it, which is the entire argument for retrieval.

Mental model

Attention is a hash table where every lookup returns a weighted blend of all entries instead of one value, and where the keys were learned rather than assigned. That single change — hard match to soft match — is what makes it differentiable, what makes it trainable end to end, and what makes it cost n² instead of n.

In practice

Problem statements that lean on this topic. Rated by the phase that makes them land.

Summarising a 400-page PDF
The window is large enough to hold the whole document; putting it there is the expensive, lossy, unrepeatable option, and the alternative is a map-reduce you already know how to build.
after phase 2 · 4 min · #context #architecture #cost

Done reading?

Nothing marks itself complete. Say so only when you could explain this to someone else.

Next: Training vs inference