Tokenization

In one line

Text becomes integers before a model sees it, and where the splits fall is learned from statistics, not from grammar.

Why it exists

A neural network multiplies matrices. It has no operation that consumes a character, so something has to turn text into numbers before the model exists at all. is that something: a deterministic, reversible function from a string to a list of integer ids.

Without it there is no input layer to build. With it, you inherit a specific consequence: the model's unit of perception is not the character and not the word, but whatever the tokenizer decided. Every cost you are billed for, every limit you hit, and every off-by-one in a prompt template is counted in those units.

The string is gone by step two. Every limit you hit downstream is measured in the boxes on the bottom row.

What a token actually is#

A is an entry in a fixed — typically 32,000 to 200,000 of them — built once before training and frozen forever after. Common English words usually get a whole entry each. Rarer strings decompose into several pieces. Punctuation, whitespace runs and newlines are entries too, which is why reformatting a prompt changes its token count.

The rough conversion for English prose is about 0.75 words per token, or four characters. That ratio is a property of the corpus the vocabulary was built from, and it degrades badly outside it. A UUID is nearly one token per two characters. Japanese and Hindi can be two or three times more expensive per word than English. Minified JavaScript is worse than formatted JavaScript, because the formatting characters were common in training and the mangled identifiers were not.

Two practical consequences follow. First, you cannot estimate cost from string length; you have to run the tokenizer, and every provider ships one for exactly this reason. Second, the tokenizer is welded to the model. Ids mean whatever the table says they mean, so feeding one model's ids to another produces confident nonsense rather than an error — the same failure mode as reading a struct with the wrong header file.

How the vocabulary gets built#

is the usual algorithm, and it is simple enough to describe in a sentence: start with every byte as its own token, find the most frequent adjacent pair in the corpus, merge it into a new token, and repeat until you hit the target vocabulary size.

Nothing in that loop knows about morphemes, syllables or parts of speech. The merges encode frequency, and frequency in the training corpus only. That is why unhappiness splits as un + happi + ness rather than the linguistically tidy un + happiness — the middle fragment was simply more common across the corpus than the clean stem.

Starting from bytes rather than characters is what makes the scheme total: any input at all, including emoji, control characters and text in scripts the corpus never contained, has some representation. It may be an expensive one, but it never fails.

Alongside the learned merges sit a handful of — reserved ids with no text of their own, marking the start of a sequence, the end of a turn, or the boundary between a system instruction and a user message. These are structural, and they are how a flat sequence of integers carries the framing of a conversation.

The math

Token count is not a function of characters, but it is bounded by them. For a vocabulary built with byte-pair encoding over a corpus with average merge depth $m$ :

n_{\text{tokens}} \approx \frac{n_{\text{bytes}}}{m}

For English prose on a typical vocabulary, $m \approx 4$ . So a 2,000-character support email is:

n_{\text{tokens}} \approx \frac{2000}{4} = 500

And the same 2,000 characters as a base64 blob, where $m \approx 1.5$ because almost no merge in the vocabulary applies:

n_{\text{tokens}} \approx \frac{2000}{1.5} \approx 1333

Same byte count, nearly triple the price and triple the consumption of your .

Worked example

Take the string user_id=42, ten characters, and run it through a typical vocabulary:

"user"      -> 1838
"_"         ->  062
"id"        -> 1300
"="         ->  028
"42"        -> 2983

Five tokens for ten characters — a ratio of two, not four, because the identifier is punctuated in ways prose is not.

Now change it to user_id=43:

"user"      -> 1838
"_"         ->  062
"id"        -> 1300
"="         ->  028
"4"         ->  019
"3"         ->  018

Six tokens. 42 was common enough in the corpus to earn its own entry; 43 was not. Two strings of identical length, differing by one character, cost different amounts. There is no rule you can apply here other than running the tokenizer.

Gotchas

Estimating tokens as len(text) / 4 and building a budget on it. The ratio holds for English prose and nothing else, and the places you are most likely to apply it — logs, JSON payloads, code — are exactly where it breaks by a factor of three. Call the provider's tokenizer, cache the count next to the content, and treat any heuristic as a lower bound.
Assuming the model can see letters. Counting the r's in a word, reversing a string, or checking whether two identifiers differ by one character are all hard for a model that never received characters in the first place. This is not a reasoning failure; the information was destroyed before the model ran. Do character-level work in code.
Trimming a prompt by characters and cutting a token in half. Truncating mid-token produces a byte sequence that re-tokenizes differently from what you intended, quietly changing the tail of your prompt. Truncate on token boundaries using the tokenizer, or truncate on a structural boundary you control, like a whole retrieved passage.
Letting user text contain the literal spelling of a special token. If your template concatenates strings rather than using the provider's message API, a user can type a role marker and change the framing of the conversation. This is the oldest form of there is, and message APIs exist precisely to make it structurally impossible.
Changing tokenizer versions without reindexing. Vocabulary updates are rare but they do happen, and cached token counts, stored embeddings and prompt-length assumptions all silently go stale. Treat the tokenizer version as part of your schema.

Mental model

The tokenizer is a compression dictionary, fitted once and shipped read-only with the model — closest in spirit to a protobuf schema. Both sides must use the same file or the bytes decode to garbage rather than to an error. And like any dictionary-based compression, it is excellent on data resembling what it was fitted to, and unremarkable to bad on everything else.

Done reading?

Nothing marks itself complete. Say so only when you could explain this to someone else.

Next: Embeddings