Encoders vs decoders

Both families are stacks of identical blocks: Attention followed by a feed-forward network, repeated dozens of times. The weights differ, the training data differs, but the arithmetic is the same.

In an Encoder, every position attends to every other position. Token three sees token nine and token nine sees token three. Representations are built with full knowledge of the surrounding text in both directions, which is exactly what you want when the whole input is available up front and the output is a judgement about it.

In a Decoder, a causal mask is added to the attention scores before the softmax: a triangular matrix of negative infinities that drives the weight of every future position to zero. Token three sees tokens one and two and nothing else. That restriction is what makes the training objective honest — the model is asked to predict token four from tokens one to three, and cannot peek — and it is what makes Autoregressive generation possible at inference time.

That is the entire architectural distinction. One mask, one bit of bookkeeping, two completely different products.

Encoders are the right tool when the output is smaller than the input: classification, entity extraction, ranking, and above all producing an Embedding for search. They run in a single forward pass whatever the output is, so latency depends only on input length. They are small — a strong retrieval encoder is a few hundred million parameters, not tens of billions — and cheap enough to run over an entire corpus on a nightly job.

Decoders are the right tool when the output is open-ended. Every chat model, every code assistant, every summariser you use is a decoder. They generate one token per forward pass, so cost and latency scale with how much they say, and they can be steered by instructions in a way an encoder cannot.

The two also appear in one system routinely. A retrieval pipeline uses an encoder to build the index and a decoder to write the answer, and the Reranking step in between is usually a third model — a Cross-encoder, which is an encoder fed the query and document together rather than separately.

A word on the third family: encoder-decoder models, where an encoder reads the input and a decoder attends to its output while generating. This was the original transformer design and it remains strong for translation and other tasks with a clean input-output pairing. Decoder-only architectures displaced it for general assistants largely because one stack is simpler to scale, not because the design was wrong.

Inside Retrieval the encoder shows up in two arrangements worth naming separately, because the choice between them is the main quality-versus-cost decision in a search stack.

A Bi-encoder runs the encoder twice, independently: once over the query and once over each document, each producing a vector, compared by distance. The documents can be embedded ahead of time and stored, so query time is one encoding plus an index lookup. This is what makes corpus-scale search possible at all.

A Cross-encoder runs the encoder once over the query and document concatenated, and emits a relevance score directly. Because attention spans both texts, it can notice that "the 2023 policy" in the query matches "prior to January 2024" in the document — a connection no pair of independently computed vectors can represent. It is also uncacheable: the score exists only for that pair, so scoring a million documents means a million forward passes.

The standard arrangement follows directly. Retrieve fifty candidates with the bi-encoder, rerank them with the cross-encoder, pass the top five to the decoder. Cheap and broad, then expensive and narrow — the same shape as any two-stage filter.

You are building support-ticket triage and need three things. Each maps to a different model family, and getting the mapping right removes most of the cost.

Route the ticket to a queue. Seven fixed categories, and you have 8,000 labelled historical tickets. A fine-tuned encoder classifier: a few hundred million parameters, roughly 15ms per ticket on a CPU, and it outputs a probability per class rather than text you have to parse. A decoder could do this, at fifty times the cost and with an output you must validate.

Find similar past tickets. Bi-encoder over every ticket, vectors in an index, nearest-neighbour lookup at query time. Embedding 8,000 tickets is a batch job measured in minutes; each lookup afterwards is a few milliseconds.

Draft the reply. Decoder, with the retrieved tickets in context. This is the only one of the three that has to be a large generative model, and it is the only one where you pay per token of output.

The failure mode is using a large decoder for all three, which is what happens when the only tool in reach is a chat API. It costs roughly two orders of magnitude more than it needs to for the first two, and the classification is less reliable because you are parsing free text where you could have had Logits.