3 · Stemming & lemmatization
Reduce word variants to a root — fast-but-ugly stemming vs accurate-but-slow lemmatization.
Reduce word variants to a single root form — stemming is fast but ugly, lemmatization is correct but slow.
Without this:
Your model treats 'running', 'ran', 'runs' as 3 different words and never learns they share meaning.
When you tokenize text, you get many inflected forms of the same base word: "run", "running", "ran", "runs". To a BOW or TF-IDF model, these are four entirely separate vocabulary entries. That's vocabulary bloat and semantic fragmentation — the model learns nothing about their shared meaning.
Two strategies to normalize to a root form:
Stemming chops off suffixes algorithmically using a set of hand-crafted rules. It's fast (O(word length)), requires no dictionary, but the result may not be a real word. "Fairly" might stem to "fairli" and "university" to "univers".
Lemmatization looks up the word in a morphological dictionary and returns the canonical base form (the lemma). "Running" (verb) → "run". "Geese" (noun) → "goose". It's slower than stemming, requires language resources (the WordNet database), and needs a POS hint to be fully accurate — "running" as a noun lemmatizes to "running", not "run".
| | Stemming | Lemmatization | |---|---|---| | Speed | Fast | Slow | | Output | May not be a real word | Always a real dictionary word | | Requires | Nothing | WordNet (or similar lexicon) | | POS-aware | No | Yes (with POS hint) | | Use when | Real-time pipelines, search indexes | Offline analytics, high-accuracy NLP |
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Stemming vs lemmatization — when to reach for each
What is the key difference between lemmatization and stemming?
- Stemming is fast and rule-based but may produce non-words (`'univers'`); lemmatization is slower but always returns real dictionary words.
- Pass a POS tag (`pos='v'` for verbs, `pos='a'` for adjectives) to `WordNetLemmatizer` for correct results — without it, everything defaults to noun.
- Neither technique is used in modern Transformer fine-tuning — attention handles morphological variation automatically.
Search engines (Elasticsearch uses Porter stemming by default); classical text classification before TF-IDF; information retrieval systems that need to match inflected forms to base queries.
If you remove it: Without normalization, 'run', 'running', 'ran', 'runs' are four separate features — your BOW/TF-IDF vectors are larger, sparser, and miss the semantic connection.