6 · One-hot encoding for text
Build a tiny OHE vocabulary by hand — then see why it explodes on real language and makes a lousy text representation.
Representing each unique word as a one-hot vector is the simplest text→vector encoding — and a useful straw man.
Without this:
You'd skip the foundational baseline that makes BOW and embeddings feel motivated.
In the previous chapter we learned how to clean and tokenize text. But ML algorithms need numbers, not strings. The first — and most naive — way to turn words into numbers is one-hot encoding (OHE).
The idea is simple: build a vocabulary of all unique words in your corpus, assign each word an integer index, then represent each word as a vector of all zeros except a single 1 at that word's index.
For a vocabulary ["cat", "dog", "feline", "pet"] (size 4):
"cat"→[1, 0, 0, 0]"dog"→[0, 1, 0, 0]"feline"→[0, 0, 1, 0]
A sentence is then represented as a sequence of OHE vectors — a 2D matrix with shape (num_tokens, vocab_size).
OHE works beautifully for categorical variables with low cardinality (colors, days of the week, zip codes). For natural language, it breaks down almost immediately — as you are about to see.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Four structural problems with OHE for text — and the techniques that fix them
What is the dimension of a one-hot vector when the vocabulary size is 50,000?
- OHE represents each word as a binary vector of length equal to vocab size — simple to build but scales catastrophically with vocabulary size.
- The four problems with OHE for text: (1) huge sparse vectors, (2) no similarity between synonyms, (3) variable-length sequences, (4) no representation for out-of-vocabulary words.
- OHE is appropriate for low-cardinality categorical features (colors, days, zip codes) — not for natural language vocabularies in the tens of thousands.
- BOW, TF-IDF, and Word2Vec each solve one or more of OHE's four problems — understanding OHE's failures is the motivation for every technique that follows.
OHE-of-tokens is a useful conceptual baseline for teaching and debugging; in production NLP it has been replaced by dense embeddings. OHE still appears for categorical input features in tabular ML (sklearn's OneHotEncoder).
If you remove it: Without understanding OHE, BOW feels arbitrary — you don't see that BOW is just 'sum the OHE rows', and embeddings feel magical rather than a principled solution to OHE's failure mode.