Phase 1 of 4
Representation
How does text become something a model can compute on?
Before any model does anything interesting, your strings have to become numbers. This phase is about that conversion and everything it quietly decides for you.
Topics in this phase
Tokenization
6 minUnreadText becomes integers before a model sees it, and where the splits fall is learned from statistics, not from grammar.
Embeddings
6 minUnreadText compressed into a fixed-length vector where geometric distance stands in for similarity of meaning.
After Tokenization
Vector space intuition
6 minUnreadDirection carries meaning and length usually does not, which is why almost every similarity measure you will use is an angle.
After Embeddings
Encoders vs decoders
6 minUnreadOne architecture reads in both directions to produce vectors, the other reads only backwards so it can write; a single mask is the difference.
After Embeddings