Training vs inference

Pretraining is gradient descent at scale, and the loop is four steps: take a batch of documents, predict the next token at every position, measure how wrong those predictions were with a loss function, and nudge every Parameter slightly in the direction that would have been less wrong. Repeat for months.

The objective really is that narrow. Not "be helpful", not "be accurate" — only "be less surprised by the token that actually came next", averaged over a corpus. Every capability a base model has is a side effect of getting good at that one thing. Perplexity is just that surprise re-expressed as a count of equally likely options.

Two things follow. First, the model learns the distribution of its corpus, including its errors and its cut-off date. It has no mechanism for knowing what it does not know, because nothing in the objective rewarded that. Second, the cost is enormous and one-time: millions of GPU-hours, which is why you are renting a model rather than training one.

A base model out of pretraining is not yet useful as an assistant — it completes text rather than following instructions. Post-training fixes that: supervised fine-tuning on instruction-response pairs, then RLHF or a similar preference method that trains a reward model on human comparisons and optimises against it. This stage is a rounding error on pretraining cost and is responsible for most of what you experience as the model's personality, willingness and refusals.

Inference is a pure function. Weights are read, never written. Nothing about request n affects request n+1.

That has direct architectural consequences, all of them familiar:

It scales horizontally without coordination. Any replica can serve any request, because there is no shared mutable state. Capacity is a memory problem, not a consistency problem.

It has no memory between calls. Conversation history feels like memory because the client resends the entire transcript on every turn. That is why long conversations cost more per message — you are paying to re-read the whole thing each time.

It is not deterministic by default, but it can be. Sampling introduces randomness; with greedy decoding and a fixed model version, the same input produces the same output, which is what makes reproducible Evals possible. Floating-point non-associativity across different batch sizes can still produce rare divergence, so treat determinism as a strong tendency rather than a guarantee.

Between "use it as it comes" and "train your own" sits a ladder of Adaptation options, ordered by cost and by how hard each is to reverse.

Prompting and examples change nothing about the model and are instant to undo. Retrieval-augmented generation adds facts at request time and is the correct answer for anything that changes — a fact updated in your database is live on the next request, with no retraining and no staleness window.

LoRA sits one rung further: freeze the original weights, train a small pair of low-rank matrices alongside them. The adapter is megabytes rather than gigabytes, several can be hot-swapped over one served base model, and quality is close to full fine-tuning for style and format work.

Full fine-tuning updates every weight. It reliably teaches form — house tone, a specific output structure, a domain's phrasing conventions — and teaches facts badly and expensively, because every new fact means another training run. The rule of thumb that survives contact with production: retrieval for what the model should know, fine-tuning for how it should behave.

Distillation is the other direction entirely: train a small model to imitate a large one's outputs on your traffic. It is the most reliable way to cut cost at fixed quality on a narrow task, and it only becomes available once you have logged enough real requests to distil from.

You are asked to make a support assistant "know" your product documentation, which changes weekly. Three options, priced honestly.

Fine-tune on the docs. A LoRA run over a few thousand examples is maybe $50 of compute and a day of work. It is also wrong: the docs change weekly, so you own a retraining pipeline forever, the model will still confidently answer from the old version, and you cannot cite a source. Facts are not what fine-tuning is for.

Retrieve. Chunk the docs, embed them, retrieve the top five passages per question, put them in the prompt. Costs a few hundred extra prompt tokens per request. An edit to a doc is live on the next query. You get citations for free. This is the right answer, and it took an afternoon.

Both. Retrieval for the facts, plus a small fine-tune so replies come out in your support team's voice with the right structure — greeting, answer, next step, no hedging. This is the real production shape, and note the split: the fine-tune is carrying form, the retrieval is carrying content.

One more number. If that assistant serves a million requests a month at 1,500 prompt tokens and 300 output tokens, prompt tokens dominate the bill by five to one. The lever on cost is prompt caching and prompt length, not the model's size — and if the task narrows enough, distillation to a small model beats both.