Turning text into numbers · Transformers

Neural networks operate on vectors; text is a string of symbols. Two steps bridge the gap: tokenization splits text into units and maps each to an integer id, and embedding maps each id to a learned vector. Then a harder problem appears: a token’s meaning depends on its context, so the model needs a way to mix information across positions. Attention is the answer to that problem.

Tokenization: choosing the units

Unit	Vocabulary size	Sequence length	Problem
Characters or bytes	tiny (256 bytes)	very long	more compute per sentence; words must be learned from scratch
Words	huge (millions)	short	unknown words, typos; “run” and “running” get unrelated ids
Subwords	32k – 256k	moderate	the standard compromise

Byte-Pair Encoding (BPE), the most common scheme, from first principles:

Start with a vocabulary of all bytes (or characters).
Count every adjacent pair of symbols in a large training corpus.
Merge the most frequent pair into a new symbol and add it to the vocabulary.
Repeat until the vocabulary reaches the target size.

Frequent words become single tokens (“the”); rare words split into pieces (“tokenization” → “token” + “ization”). Byte-level BPE, used since GPT-2, starts from raw bytes, so any string in any language, emoji included, is representable and there is no “unknown” token.

Variants: WordPiece (BERT) merges by likelihood gain instead of raw frequency; Unigram LM (in the SentencePiece library) starts with a big vocabulary and prunes it. SentencePiece treats the space as an ordinary symbol (shown as ▁), which makes tokenization language-independent.

Why tokenization matters in systems work

Cost and context are counted in tokens, not words. English averages roughly 0.75 words per token; German compounds and non-Latin scripts usually need more tokens per word, so the same content costs more.
Token counts are not portable. Each model family has its own tokenizer; measure with the tokenizer of the model you actually use.
The model never sees characters. “strawberry” may be two or three opaque tokens, so spelling, letter counting, string reversal and digit-level arithmetic are hard. How numbers are split (digit by digit or in chunks) visibly affects arithmetic.
Whitespace matters. “ Hello” and “Hello” are different tokens; a stray trailing space at the end of a prompt can degrade output.
Special tokens (beginning of sequence, end of turn, role markers) structure chat formats (Part 11). User text that contains them must be handled carefully.

Embeddings: ids to vectors

An embedding layer is a lookup table E of shape (V, d): token id i fetches row i. Mathematically it is a one-hot vector times a matrix, so it is just a linear layer trained by backprop like everything else. Training places tokens used in similar contexts near each other: you know a word by the company it keeps.

At the output, the unembedding matrix maps a d-dimensional vector to V logits, one score per vocabulary token. Many models tie it to the input embedding (weight tying), saving V × d parameters; for V = 128k and d = 4096 that is about 0.5B parameters.

The sequence problem

After embedding, “bank” has the same vector in “river bank” and “bank account”. The model needs contextual representations, which means mixing information across positions. The requirements:

handle input of any length;
connect tokens that are far apart;
train in parallel over all positions, because GPUs are only fast at large parallel matrix multiplies.

Recurrent networks: the previous answer

An RNN reads tokens one at a time, updating a hidden state:

h_t = \tanh(W h_{t-1} + U x_t + b)

Three structural problems:

No parallelism over time. h_t needs h_{t−1}, so a 1,000-token sequence takes 1,000 sequential steps and wastes GPU parallelism.
Vanishing or exploding gradients. Backprop through time multiplies by W at every step. LSTMs and GRUs add gates and an additive cell state (a residual idea) that help but don’t remove the problem.
A fixed-size bottleneck. Everything about the past must fit into one vector h_t.

In 2014-era encoder–decoder translation, the entire source sentence was compressed into one vector before decoding: the bottleneck at its worst. Bahdanau et al. let the decoder look back at all encoder states at every step and take a weighted average, with weights computed from relevance. That was attention, bolted onto an RNN. In 2017, “Attention Is All You Need” asked what happens if attention is the only mechanism, with no recurrence at all.

The comparison that motivated transformers

Layer type	Compute per layer	Sequential steps	Path between distant tokens
Recurrent	O(T · d²)	O(T)	O(T)
Convolution (kernel k)	O(k · T · d²)	O(1)	O(log_k T)
Self-attention	O(T² · d)	O(1)	O(1)

Self-attention connects any two tokens in one step and parallelizes completely across positions. The price is the T² term: cost grows with the square of sequence length (Parts 4, 9 and 10). The table adapts one from the 2017 paper.

Questions you should now be able to answer

Walk through the first three BPE merges on the corpus “low lower lowest”.
Why might a model miscount the r’s in “strawberry”, and what would a byte-level model change?
Why do the embedding and unembedding matrices have related shapes, and what does tying them save?
What are the three structural limits of RNNs, and which one did attention fix first?
In the comparison table, where does attention’s advantage come from, and what does it cost?