8 · N-grams (bigrams & trigrams)
Recover partial word order by treating token pairs and triples as features — so 'not good' and 'good' are no longer the same.
BOW loses word order — n-grams partially recover it by treating consecutive token pairs and triples as features.
Without this:
'not good' and 'good not' have identical BOW representations — n-grams catch the difference.
BOW treats each word independently. That's usually fine — but in some tasks the relationship between adjacent words carries critical information. The word "not" on its own means little; "not good" as a pair means the opposite of "good".
An n-gram is a contiguous sequence of n tokens:
- Unigram (n=1): each individual token — the standard BOW feature
- Bigram (n=2): every consecutive pair — "not good", "highly recommend", "machine learning"
- Trigram (n=3): every consecutive triple — "not at all", "state of the art"
sklearn's CountVectorizer supports n-grams natively via the ngram_range=(min_n, max_n) parameter. Setting ngram_range=(1, 2) includes both unigrams and bigrams, giving you richer context without discarding the raw word counts.
The tradeoff: vocabulary size grows quickly. A corpus with 1,000 unique words could have up to 1,000,000 possible bigrams — most of which appear rarely or never. That's why min_df (minimum document frequency) is your best friend with n-grams.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
N-gram range tradeoffs and practical guidelines
If a vocabulary has 1,000 unique unigrams, how many bigrams could theoretically exist?
- `CountVectorizer(ngram_range=(1, 2))` includes both unigrams and bigrams — the most common sweet spot that adds negation/phrase context without exploding vocabulary too much.
- Bigrams catch critical sentiment signals like 'not good', 'barely acceptable', 'highly recommend' that are invisible to unigram BOW.
- Vocabulary grows as k² for bigrams and k³ for trigrams — always use `min_df` cutoffs to prune rare n-grams.
- N-grams recover LOCAL order within a window; self-attention in Transformers recovers ALL-PAIRS global order — conceptually n-grams are the ancestor.
Classical text classification leaderboards (Kaggle sentiment, news categorization) almost always had bigram + TF-IDF in the top solutions. Modern Transformers replace n-grams with self-attention's all-pairs interactions — but understanding n-grams makes attention mechanisms intuitive.
If you remove it: Without n-grams, negation-heavy text (reviews, sentiment) is much harder for classical models. 'Not good' and 'good' become indistinguishable features.