37 · Subword tokenization
Word-level vocabularies explode and fail on unknown words; character-level makes sequences too long. Subword tokenization (BPE) is the middle ground that powers GPT and BERT.
Subword tokenization (BPE) starts from characters and iteratively merges the most frequent adjacent pair, giving a fixed, manageable vocabulary with NO out-of-vocabulary failures and sequences far shorter than character-level.
Without this:
Every LLM you use first turns text into subword token IDs. Without understanding tokenization you can't reason about context limits, pricing, or why models stumble on spelling tasks.
Before a transformer can do anything, raw text must become a sequence of integer IDs. How we split text into those units — tokenization — is a design choice with deep consequences, and the obvious options both fail.
Word-level tokenization assigns one ID per word. English has hundreds of thousands of word forms, so the vocabulary (and the embedding table) explodes. Worse, any word NOT in the vocabulary — a typo, a new product name, a rare technical term — becomes a single <UNK> "out-of-vocabulary" (OOV) token, and all its meaning is lost. You can never enumerate every word a model will meet.
Character-level tokenization swings the other way: the vocabulary is tiny (~100 symbols) and there is no OOV problem, since any word is just a sequence of characters. But now a 10-word sentence becomes 50+ tokens. Sequences get very long, attention cost grows quadratically with length, and the model must work harder to assemble meaning from individual letters.
Subword tokenization is the middle ground that powers GPT, BERT, and essentially every modern LLM. Common words stay whole ("the", "running"), while rare words split into meaningful pieces ("tokenization" → "token" + "ization"). The vocabulary is fixed and manageable (typically 30k–100k), there are NO out-of-vocabulary failures (worst case, a word falls back to characters), and sequences stay far shorter than character-level.
The classic algorithm is Byte-Pair Encoding (BPE). It starts by splitting every word into individual characters plus an end-of-word marker </w> (so the tokenizer knows where words end). Then it repeatedly finds the most frequent adjacent pair of symbols across the corpus and merges them into a new symbol, recording the merge. After a fixed number of merges, the learned merge list IS the tokenizer. Frequent character sequences naturally fuse into whole subwords; rare ones stay split.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Subword tokenizer families: WordPiece (likelihood + ## prefix), SentencePiece (language-agnostic, raw bytes), tiktoken (byte-level BPE), and special tokens.
Why do LLMs use subword tokenization instead of word-level or character-level?
- Word-level tokenization explodes the vocabulary and fails on out-of-vocabulary words; character-level avoids OOV but makes sequences too long.
- BPE starts from characters + an end-of-word marker </w> and repeatedly merges the most frequent adjacent pair; the ordered merge list IS the tokenizer.
- Variants differ in the merge criterion or alphabet: WordPiece (likelihood, ## prefix), SentencePiece (raw bytes, language-agnostic), tiktoken (byte-level BPE — zero OOV).
Every prompt you send to GPT, Claude, or BERT is first run through a subword tokenizer. Token counts drive API pricing, context-window limits, and explain spelling/counting failures.
If you remove it: You'd treat the tokenizer as a black box, unable to explain why rare words cost more tokens, why context limits are in tokens, or why 'count the letters' tasks fail.