Glossary

92 terms. Each one is capped at 80 words — anything longer is a subtopic inside a page, not an entry here. Terms that also have a full topic link through to it.

A

full topic ↗: Any change that makes a general model fit your task: prompting, examples, retrieval, adapters, full fine-tuning. They form a ladder ordered by cost and by how hard each is to undo, and the discipline is to stop climbing as soon as the evals stop improving.
full topic ↗: A loop in which a model chooses a tool, your code runs it, the result goes back into context, and the loop repeats until a stopping condition. The interesting engineering is entirely in the loop — the budgets, the retries, the halting rule — not in the model.
: Observe, decide, act, repeat. Every iteration appends to the context, so cost grows superlinearly with steps and the transcript eventually crowds out the original instruction. A hard step budget is not a nicety; it is the only thing bounding spend.
: Search that returns probably-the-closest vectors instead of certainly-the-closest, in exchange for orders of magnitude less work. The approximation is the point: exact search over millions of high-dimensional vectors is a full scan, and no index avoids that while staying exact.
full topic ↗: The mechanism by which one token's representation is updated using a weighted blend of others. Weights come from dot products between learned query and key vectors, so the model decides what is relevant rather than being told. Cost grows with the square of sequence length.
: One independent attention computation inside a layer. A layer runs many in parallel over slices of the vector and concatenates the results, so different heads can specialise — one tracking syntax, another tracking a subject across a long passage — without competing for the same weights.
: Generating a sequence one element at a time, conditioning each on everything produced so far. It is why output cannot be parallelised within one request, why streaming is natural, and why a mistake early in a response tends to be built on rather than corrected.

B

: How many requests are decoded together in one pass. Larger batches amortise the weight read over more work, raising throughput until KV cache memory runs out. It is the main dial between cost per token and latency per request.
: Two passes of the same encoder, one over the query and one over each document, compared by vector distance. Documents can be embedded once and indexed, so search is a nearest-neighbour lookup. Fast and scalable, and less accurate than letting the model see both texts together.
: The standard keyword ranking function, built on term frequency and document length. Decades old, extremely fast, and still hard to beat when a query contains an exact token — an error code, a product SKU, a surname — that no embedding places meaningfully.
: The usual algorithm for building a vocabulary. Start from raw bytes, repeatedly merge the most frequent adjacent pair, stop at the target size. The merges are learned from a corpus, so the split points reflect statistics rather than grammar — which is why "unhappiness" splits where it does.

C

: A triangular matrix of negative infinities added to attention scores so that position n contributes nothing to any earlier position. It is one cheap piece of bookkeeping, and it is the entire architectural difference between a model that reads and a model that writes.
: Repeating a slice of text at the boundary between consecutive chunks so an idea split across a break still appears whole somewhere. Cheap insurance against arbitrary split points, paid for in index size and in near-duplicate results that a reranker then has to collapse.
: Splitting documents into passages small enough to retrieve precisely and large enough to stand alone. Too small and a chunk loses the context that made it meaningful; too large and one relevant sentence drags four irrelevant paragraphs into the prompt with it.
: A switch that stops calling a failing dependency and fails fast instead. Worth naming separately here because the failure being tripped on is often a budget or a loop count rather than an error rate — an agent that is not erroring can still be burning money.
: Masking out every token that would make the output violate a grammar or schema, at each step. Since illegal tokens have zero probability of being chosen, the result is valid by construction rather than by luck. Validity is guaranteed; usefulness is not.
full topic ↗: Deciding what occupies the context window on each call, and in what order. Instructions, retrieved passages, history and tool output all compete for the same finite space, and the discipline is mostly about what to leave out.
full topic ↗: The maximum number of tokens a model can attend over in one call, counting the prompt and the generation together. It is a hard architectural ceiling, not a soft budget. Everything in context competes for the same space, which is what makes deciding what goes in it an engineering problem.
full topic ↗: Letting new requests join an in-flight batch at each decode step instead of waiting for the whole batch to finish. Since decoding is bandwidth-bound, one weight read serves every request in the batch, and throughput rises several-fold with little latency cost.
: The cosine of the angle between two vectors: 1 means same direction, 0 means unrelated, -1 means opposite. It ignores vector length, which matters because length often encodes how common a phrase is rather than what it means. The default similarity measure for text embeddings. $\cos(\theta) = \frac{a \cdot b}{\lVert a \rVert \lVert b \rVert}$
: One pass over the query and document concatenated, producing a relevance score directly. It sees both texts at once so it is markedly more accurate than a bi-encoder, and it cannot be precomputed. The standard use is reranking the top fifty candidates, never scanning the corpus.

D

full topic ↗: Generating one token, then feeding it back in to generate the next. Each step reads every weight in the model to produce a single token, which makes decoding memory-bandwidth-bound rather than compute-bound and explains why output length dominates latency.
full topic ↗: A stack where each position can only attend to positions before it, enforced by a mask. That restriction is what makes generation possible: the model can be trained to predict the next token without ever seeing it. Every chat model you use is a decoder.
: How many numbers are in each vector — commonly 384 to 4,096. More dimensions can carry more distinctions but cost proportionally more memory, more bandwidth and more index size. It is the same trade as picking a hash width: bigger is not free and rarely the binding constraint.
: Training a small model to imitate a large one's outputs. The small model ends up much better at the specific distribution it was shown than its size would suggest, and no better anywhere else — which is exactly the trade you want for a narrow, high-volume task.
: Multiply two vectors element by element and add the results. It is the single arithmetic operation underneath both similarity search and attention, which is why hardware that does it fast defines what models are practical. For unit-length vectors it equals cosine similarity exactly. $a \cdot b = \sum_i a_i b_i$
: Behaviour changing while your code did not — a provider updated a model, your corpus grew, user phrasing shifted. It is the reason evals run on a schedule rather than only on deploy, and the reason pinning a model version is worth the upgrade debt.

E

full topic ↗: A vector of floating-point numbers standing in for a piece of text, where geometric closeness approximates similarity of meaning. Token embeddings are looked up per token; sentence embeddings are produced by running a model over a whole passage. Both are just arrays, and comparing them is arithmetic.
: A table with one row per vocabulary entry and one column per dimension. Turning a token id into a vector is a row lookup, not a computation. The table is learned during training and is often a large share of a small model's total parameter count.
full topic ↗: A stack where every token attends to every other token in both directions. It reads the whole input at once and produces representations, not text. Encoders are what you want for classification, retrieval and ranking — anything where the output is a decision or a vector.
: Straight-line distance between two points in vector space. It is sensitive to magnitude as well as direction, so two passages about the same subject can land far apart simply because one is longer. Usually the wrong default for text, and identical to cosine once vectors are normalised.
full topic ↗: An automated test for a non-deterministic component: fixed inputs, graded outputs, a score you can track across changes. Without one, every prompt change is a guess, because a model's response to an unchanged input is not itself stable.

F

: The per-token transformation that follows attention in every block, typically two-thirds of all parameters. Attention decides what to look at; the feed-forward network does most of the actual processing on each position independently, with no communication between tokens.
: Including two to five worked examples in the prompt so the model infers the pattern. It is the cheapest way to pin down output shape and edge-case handling, and it beats a paragraph of description almost every time. The examples cost tokens on every call.
full topic ↗: Continuing training on a smaller, targeted dataset to shift a model's behaviour. It reliably teaches form — tone, format, a house style — and is a poor and expensive way to teach facts, which belong in retrieval. Each new fact would otherwise mean another training run.
: Floating-point operations, the unit of compute cost. A useful rule: one forward pass costs roughly two FLOPs per parameter per token. It tells you what prefill costs, and it is the wrong lens for decoding, which is bound by memory traffic instead.

G

: A curated collection of inputs with known-good outputs, ideally drawn from real traffic and including the cases that have already broken once. Fifty carefully chosen examples find more regressions than a thousand synthetic ones, and take an afternoon to assemble.
: The optimisation loop behind training: measure how wrong the output was, compute which direction each parameter should move to be less wrong, take a small step, repeat. Millions of times. Nothing about it is specific to language — it is the same procedure used to fit any differentiable model.
: Always take the highest-probability token. Deterministic for a fixed model and input, which makes it the right choice for extraction and classification, and a poor one for prose, where it produces flat and repetitive text.
: A deterministic check around the model rather than inside it: schema validation, allowlists, spend caps, a human approval step on destructive actions. Guardrails are ordinary code, and they are the only part of the system whose behaviour you can actually guarantee.

H

full topic ↗: Fluent output that is not supported by anything the model was given. It is not a bug to be patched but the default behaviour of a system trained to produce plausible continuations. The engineering answer is grounding and citation, not stronger instructions.
: Hierarchical navigable small world — a layered proximity graph where search starts on a sparse top layer and descends, following edges toward the query. Fast and accurate, memory-hungry, and awkward to update in bulk. The default index in most vector databases.
: Running keyword and vector search together and merging the rankings. They fail in different directions: keyword search misses paraphrase, vector search misses exact identifiers and rare terms. Combining them costs one extra query and fixes most of the embarrassing misses.

I

: The property that running an operation twice has the same effect as running it once. It matters more here than in ordinary services because a retried or duplicated tool call is normal rather than exceptional — the model may simply decide to call it again.
full topic ↗: Running a trained model forward to get an output. Weights are read, never written, so inference is stateless between requests and scales horizontally like any other read path. Everything you deploy is inference; training is somebody else's batch job.
: The gap between successive output tokens once generation has started, typically steady for a given model and load. Multiply it by expected output length to get the rest of the response time. Together with time to first token it fully describes streaming latency.

J

: The vocabulary for describing the shape of a JSON object — fields, types, enumerations, which keys are required. It is how you tell an API what structure to enforce, and the same document doubles as the validator on your own side of the boundary.

K

full topic ↗: Per-request storage of the key and value vectors already computed, so each new token attends over history without recomputing it. It converts quadratic regeneration into linear growth, and it is usually the memory that limits how many requests fit on one GPU.

L

: Dividing a vector by its own length so it sits on the unit sphere. After this step dot product, cosine similarity and euclidean ranking all agree, which lets an index use whichever is fastest. Do it once at write time rather than on every query.
full topic ↗: One repeated block — attention followed by a feed-forward network — applied in sequence. Models are dozens of identical blocks stacked, so depth multiplies latency directly. Nothing about layer forty differs structurally from layer two; only the weights differ.
: Using a model to grade another model's output against a rubric. It scales to open-ended tasks where no exact answer exists, and it inherits its own biases — toward length, toward its own phrasing. Calibrate it against human labels before trusting the number.
: The raw, unnormalised score the model assigns to every token in the vocabulary at one position. Sampling parameters operate on these before softmax turns them into probabilities. Constrained decoding works by setting the logits of illegal tokens to negative infinity.
: Low-rank adaptation: freeze the original weights and train a small pair of matrices alongside them. Adapters are megabytes rather than gigabytes, several can be swapped over one served base model, and quality is close to full fine-tuning for most style and format tasks.
: The single number training tries to minimise. For language models it is almost always how surprised the model was by the token that actually came next, averaged over a batch. Every capability a model has is a side effect of getting good at that one objective.

M

: How fast weights can be moved from GPU memory into the compute units. Since every decode step reads the whole model to emit one token, bandwidth rather than arithmetic sets the ceiling on generation speed. This is why batching helps so much: the same read serves many requests.

N

: How many bits each stored number gets. Training usually needs 16 or 32 bits; inference often tolerates 8 or 4. Precision is a per-tensor choice, not a global one, which is why mixed schemes that keep sensitive layers wide are the norm.

O

: What you record to know the system is healthy. Alongside the usual latency and error rates, LLM systems need token counts, retrieval recall, refusal rate and output-length distribution — the leading indicators that move well before user complaints do.

P

full topic ↗: One learned number inside the model. Parameter count sets memory footprint almost directly: at 16-bit precision, weights alone need roughly two bytes each, so an eight-billion-parameter model needs about sixteen gigabytes before any activations or cache.
: The exponential of average loss — roughly, how many equally likely options the model felt it was choosing between at each token. Useful for comparing two checkpoints of the same model, nearly meaningless across model families, and no substitute for a task eval.
: Collapsing one vector per token into a single vector for the whole passage, usually by averaging or by taking a designated token's output. The choice changes the resulting geometry, so an index built with mean pooling cannot be queried with vectors produced some other way.
full topic ↗: Processing the whole input prompt in one parallel pass to populate the cache before any output appears. It is compute-bound and scales with prompt length, so it sets time to first token. Doubling the prompt roughly doubles the wait before the first character.
full topic ↗: The long, expensive pass over a very large corpus that produces a base model. Measured in millions of GPU-hours, run once, and the reason you are renting a model rather than training one. Everything after it is comparatively cheap adjustment.
: Reusing the computed KV cache for a prompt prefix that has already been seen, so the repeated part skips prefill entirely. It rewards putting stable content first and volatile content last — the same instinct as ordering a Dockerfile for layer caching.
: Content that reaches the context window and issues instructions of its own — in a retrieved document, a web page, a tool result. The model cannot reliably tell data from instruction, so the defence is to treat every model output as untrusted input at the point it triggers an action.

Q

full topic ↗: Storing weights at lower numeric precision — 8-bit or 4-bit instead of 16 — to cut memory and bandwidth. Smaller weights move faster, so it usually speeds decoding as well as shrinking the footprint. Quality loss is small but real and concentrated in edge cases.
: Three projections of each token. The query asks what this position is looking for, the key advertises what a position offers, and the value is what actually gets mixed in. Scores come from query-key dot products; the output is a weighted sum of values. It is a soft lookup table.

R

: The fraction of genuinely relevant results the retrieval step actually returned. It is the metric that matters for RAG, because the generator cannot use a passage it never saw. A system can have excellent answers on every query it retrieves correctly and still be unusable.
: The evals that run on every change, gating deploys the way unit tests do. The distinguishing feature is a tolerance: scores fluctuate, so the gate is a threshold and a trend rather than an exact match, and a flaky eval is worse than no eval at all.
: A second, more expensive scoring pass over the top few dozen candidates from a cheap first stage. Because it only sees a shortlist it can afford a cross-encoder, and it typically improves the quality of what reaches the prompt more than any embedding upgrade.
: The running vector each layer reads from and adds back into, rather than replacing. Think of it as a bus every component writes onto. It is why very deep stacks train at all, and why interpretability work talks about reading information off it.
full topic ↗: Fetching relevant text at request time and putting it in the prompt, rather than hoping the model memorised it. It is the answer to facts that change, facts that are private, and facts that need a citation — which is most facts a business cares about.
full topic ↗: The pattern of retrieving passages, putting them in the context window, and asking the model to answer from them. Most of the engineering is in the retrieval half; the generation half is a prompt. When RAG systems fail, it is almost always the search that failed.
: Reinforcement learning from human feedback — training a reward model on human preferences between outputs, then optimising the language model against it. It is what turns a raw next-token predictor into something that follows instructions and declines requests.

S

: Serving a stored answer when a new question is close enough in embedding space to an old one. The hit rate is attractive and the failure mode is severe: "close enough" is a similarity threshold, and two questions can be near-identical in wording with opposite correct answers.
: Turns a list of arbitrary numbers into a probability distribution that sums to one, exaggerating differences along the way. It appears twice per model: once converting attention scores into weights, and once converting final logits into next-token probabilities.
: One timed unit of work within a trace: a retrieval, a model call, a tool execution. Spans nest, so an agent's sixth iteration is a child of the loop rather than an unrelated event, and cost can be attributed to a branch rather than a request.
: A reserved id with no text of its own, used as structure — start of sequence, end of turn, role boundaries in a chat template. They are how a flat token stream carries framing. Sending raw text that happens to contain their literal spelling is the oldest injection trick there is.
: Have a small fast model draft several tokens, then have the large model verify them in one parallel pass. Accepted drafts are free; rejected ones cost a rollback. Output is identical to normal decoding, which is what makes it a pure latency optimisation.
full topic ↗: Getting a machine-readable object back rather than prose, so downstream code can consume it without parsing English. Modern APIs enforce a schema during decoding, which turns "usually valid JSON" into "always valid JSON" and removes an entire class of retry logic.
: A token that is part of a word rather than a whole one. Subwords are what let a fixed vocabulary cover text it has never seen: an unknown name still decomposes into pieces that exist. The cost is that rare words, code identifiers and non-English text burn far more tokens than they look like they should.
: Standing instructions placed before the conversation, defining role, format and limits. It is a strong prior, not an enforcement mechanism: anything genuinely security-relevant belongs in code around the model, because text in a prompt can always be argued with by other text.

T

: The slow end of the distribution, p95 and p99. It is worse here than in ordinary services because response time scales with output length, so the tail is partly a property of what the model decided to say. Cap output tokens to cap the tail.
: A divisor applied to logits before softmax. Below one it sharpens the distribution toward the most likely token; above one it flattens it. Zero is effectively greedy. It is a diversity knob, not a correctness knob, and turning it down does not make a model truthful.
: Tokens produced per second across all concurrent requests — the number that sets cost per token. It moves in the opposite direction to per-request latency under load, which is why one capacity target cannot serve both an interactive product and a batch pipeline.
: How long between sending a request and the first character arriving, dominated by prefill and by queueing. It is the latency number users actually feel in a streaming interface, and it is largely independent of how long the full response turns out to be.
full topic ↗: The unit a model actually reads and writes. Usually a few characters — a common word is one token, a rare one splits into several. Every limit and every price you deal with is counted in tokens, so "how long is this string" is always the wrong question and "how many tokens" is the right one.
full topic ↗: The deterministic function that turns a string into a list of integer ids, and back again. It is trained separately from the model and frozen with it, so a model and its tokenizer are a matched pair. Swapping one for another silently produces nonsense rather than an error.
full topic ↗: Giving the model a set of function signatures it can request calls to, and executing those calls yourself. The model never runs anything — it emits a name and arguments, your code decides whether to comply. That boundary is where all the authorisation belongs.
: Sample only from the smallest set of tokens whose probabilities sum to p, discarding the long tail. Unlike a fixed top-k it adapts: a confident position keeps few candidates, an uncertain one keeps many. Commonly paired with temperature.
full topic ↗: Recording the full chain for one request — prompt, retrieved passages, tool calls, raw output, token counts, latency — as linked spans. Aggregate metrics tell you something regressed; only a trace tells you which of the six stages did it.

V

full topic ↗: A data structure for finding the nearest vectors to a query without comparing against all of them. It trades exactness for speed, so unlike a B-tree it can legitimately miss a match. That trade is configurable and needs to be measured, not assumed.
: The fixed set of tokens a tokenizer can emit, typically 32,000 to 200,000 entries. Larger vocabularies mean fewer tokens per sentence but a bigger embedding matrix and a bigger output layer. It is a compression ratio traded against memory, decided once before training and unchangeable afterwards.