AI fundamentals · Phase 1 of 4
Representation
How does text become something a model can compute on?
Before any model does anything interesting, your strings have to become numbers. This phase is about that conversion and everything it quietly decides for you.
Topics in this phase
Tokenization
6 minUnreadText becomes integers before a model sees it, and where the splits fall is learned from statistics, not from grammar.
Embeddings
6 minUnreadText compressed into a fixed-length vector where geometric distance stands in for similarity of meaning.
After Tokenization
Vector space intuition
6 minUnreadDirection carries meaning and length usually does not, which is why almost every similarity measure you will use is an angle.
After Embeddings
Encoders vs decoders
6 minUnreadOne architecture reads in both directions to produce vectors, the other reads only backwards so it can write; a single mask is the difference.
After Embeddings
Examples that open after this phase
Problem statements that make sense once these topics are done. They unlock at the same 80% mark the next phase does.
Why it cannot count the r's in "strawberry"
The letters were destroyed before the model ran — and once you see that, a whole family of "the AI is bad at simple things" complaints becomes one bug with one fix.
after phase 1 · 3 min · #tokenization #debugging #mindset
Finding duplicate tickets that share no words
Two tickets say the same thing in different words; embeddings put them nearly on top of each other, and the only real decision left is where you draw the line.
after phase 1 · 4 min · #embeddings #search #product
Replacing keyword search on the internal wiki
Vector search fixes the synonym problem and quietly breaks exact lookups; the answer is not to pick one, because each misses in a direction the other catches.
after phase 1 · 4 min · #search #embeddings #retrieval
The invoice is three times the estimate
The estimate used "four characters per token"; the workload was JSON and identifiers, where the real ratio is closer to two — and the only reliable estimator is the tokenizer itself.
after phase 1 · 4 min · #tokenization #cost
Routing tickets with the wrong kind of model
A large chat model can sort tickets into queues, at two hundred times the cost of a small classifier that does it better — because the output is a label, not a sentence.
after phase 1 · 4 min · #classification #cost #architecture
Why the Hindi support queue costs three times more
The tokenizer was fitted to English, so every other language pays a surcharge in tokens, context and latency — a fairness problem hiding inside a cost line.
after phase 1 · 4 min · #tokenization #cost #product