Embeddings

In one line

Text compressed into a fixed-length vector where geometric distance stands in for similarity of meaning.

Why it exists

Integer token ids are useless for computation. Id 1838 is not meaningfully larger than id 1300, and nothing about their numeric distance says anything about their meaning. Arithmetic on ids is arithmetic on arbitrary labels.

An fixes this by replacing each id with a vector of floats, chosen during training so that vectors for things used in similar ways end up pointing in similar directions. That single property — meaning becomes geometry — is what makes semantic search, clustering, deduplication and retrieval possible with nothing more exotic than a dot product.

Turning a token into a vector is a table lookup. The table is a parameter, which is why two models never agree on coordinates.

Token embeddings and text embeddings#

Two different things wear the same name, and conflating them causes real bugs.

A token embedding is one row of the , a table with one row per vocabulary entry. Converting a id to its vector is a row lookup — no computation, just an index into an array. That table is a learned of the model, often a substantial fraction of a small model's total size.

A text embedding is one vector for a whole passage, produced by running an over the text and then the per-token outputs into a single vector, usually by averaging. This is what a vector database stores. It is context-sensitive in a way a token embedding is not: the same word in two sentences contributes differently depending on what surrounds it.

The practical distinction is that token embeddings are an implementation detail of the model and text embeddings are an artefact you own, store and index. When someone says "we embedded the documents", they mean the second.

Dimensionality and what it buys#

— the length of the vector — usually runs from 384 to 4,096. More dimensions give the model more room to keep distinctions apart, and cost you proportionally: storage, memory bandwidth during search, and index build time all scale with it.

The instinct that more is better is usually wrong in practice. Going from 768 to 3,072 dimensions quadruples your index memory and typically moves retrieval quality by a few percent, which is smaller than the gain from fixing your strategy or adding a reranking pass. Dimensionality is rarely the binding constraint, and it is the most expensive knob to change later because changing it means re-embedding the entire corpus.

Several modern embedding models are trained so that a prefix of the vector is itself a usable embedding, which lets you store 1,536 dimensions and search over the first 256 for a cheap first pass. If your provider supports it, it is close to free throughput.

Why two models never agree#

An embedding has no absolute meaning. The coordinates are whatever the training run happened to settle on, and a different run of the same architecture on the same data would produce a different, equally valid space. Only relative positions within one space carry information.

This has a hard operational consequence: vectors from two different models are not comparable, and mixing them in one produces results that are not merely worse but meaningless. There is no conversion. An embedding model upgrade is a full corpus re-embed and a full index rebuild, which is why the model identifier belongs in your index metadata from day one.

The same logic applies to how you pool, whether you normalise, and what prefix instruction you prepend — several models expect query: and passage: markers and produce subtly different geometry without them. Write-time and read-time must agree exactly.

The math

Similarity is the cosine of the angle between two vectors — direction only, ignoring length:

\cos(\theta) = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}

With $a = [1, 2, 3]$ and $b = [4, 5, 6]$ :

\cos(\theta) = \frac{4 + 10 + 18}{\sqrt{14}\,\sqrt{77}} = \frac{32}{32.83} = 0.975

Three multiplications, two additions, two square roots. That is the whole of , and the whole of semantic search once you have an index that avoids doing it a million times per query.

Worked example

Suppose you embed three short strings with the same model and get back three-dimensional vectors (real ones have hundreds more, but the arithmetic is identical):

"reset my password"   ->  [0.81, 0.12, 0.57]
"I forgot my login"   ->  [0.78, 0.19, 0.59]
"what is your refund policy"
                      ->  [0.11, 0.93, 0.34]

Cosine between the first two:

dot   = 0.81(0.78) + 0.12(0.19) + 0.57(0.59) = 0.988
|a|   = 0.998    |b|  = 0.993
cos   = 0.988 / (0.998 x 0.993) = 0.997

Cosine between the first and third:

dot   = 0.81(0.11) + 0.12(0.93) + 0.57(0.34) = 0.395
cos   = 0.395 / (0.998 x 1.00) = 0.396

0.997 against 0.396. The two phrasings of the same request share almost no words, and the model has placed them nearly on top of each other. That gap is the entire value proposition of embeddings over keyword matching.

Gotchas

Treating a similarity score as a probability. 0.87 does not mean 87% confident, and the useful threshold differs per model and per corpus — some models put everything between 0.7 and 0.95. Calibrate against your own labelled pairs, and prefer ranking over thresholding wherever you can.
Mixing vectors from different models, versions or pooling strategies in one index. There is no error and no exception, just steadily worse results that look like a relevance problem. Store the model id and dimension alongside every vector and refuse writes that disagree.
Embedding a 40-page document as one vector. Averaging over that much text produces something near the centroid of the whole corpus — similar to everything, useful for nothing. Chunk first, embed the chunks, and retrieve at chunk granularity.
Using a where you needed a . Two separately embedded texts compared by distance is fast but coarse. If the top ten results are nearly right but badly ordered, the fix is a reranking pass over the shortlist, not a bigger embedding model.
Building a on a fixed similarity threshold. "How do I cancel?" and "How do I cancel my cancellation?" sit very close together and have opposite answers. Caching on near-duplicates is a correctness decision dressed as a performance one.

Mental model

An embedding is a lossy hash with a useful defect: similar inputs produce similar outputs. A cryptographic hash is engineered so that one flipped bit scatters the output — that is the point of it. An embedding is engineered for exactly the opposite, so that nearby inputs land nearby and you can ask about neighbourhoods instead of exact matches. Everything else about it — the storage, the index, the version pinning — behaves like a hash you have to keep in sync with the function that produced it.

Done reading?

Nothing marks itself complete. Say so only when you could explain this to someone else.

Next: Vector space intuition