4 · Stopwords, lowercasing & punctuation
Strip high-frequency low-information words so your TF-IDF and BOW vectors carry real signal.
Strip the high-frequency low-information words ('the', 'is', 'and') and your TF-IDF / BOW vectors get tighter and faster.
Without this:
Stopwords dominate your token counts and drown the actual signal.
Stopwords are extremely common words that carry little standalone semantic meaning — "the", "is", "at", "on", "and", "a". In a Bag of Words model, these words would appear in almost every document and therefore provide almost no discriminative signal. They inflate vocabulary size and computing costs without helping the model learn anything useful.
Combined with lowercasing (so "The" and "the" map to the same token) and punctuation removal (so only alphabetic tokens survive), stopword filtering is typically the most impactful single preprocessing step for classical NLP.
When NOT to remove stopwords:
The classic exception is sentiment analysis. Consider: "This product is NOT good." Remove "not" as a stopword and you get "product good" — the exact opposite sentiment. For sentiment tasks, negation words ("not", "never", "barely", "hardly") are critical signal and must be preserved.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Stopword removal decision guide — task-dependent
For a sentiment classifier working on 'this product is NOT good', should you remove the word 'not'?
- NLTK provides 179 English and 313 Spanish stopwords; always audit these against your domain — generic lists miss domain-specific high-frequency low-signal words.
- The standard classical preprocessing one-liner: `[t for t in word_tokenize(text.lower()) if t.isalpha() and t not in stop_words]`.
- For sentiment analysis, negation words ('not', 'never') must be kept — they flip the polarity of the sentence.
- Modern Transformers skip stopword removal entirely — attention weights handle the job dynamically.
Classical retrieval and classification — Lucene/Elasticsearch removes stopwords by default to speed up index lookups; sklearn's `CountVectorizer` and `TfidfVectorizer` accept a `stop_words='english'` parameter.
If you remove it: Without stopword removal, 'the', 'is', 'a' become the most common tokens in every document — BOW vectors are dominated by noise and TF-IDF weights for content words collapse.