17 · Spam classifier: BOW vs TF-IDF comparison
Swap CountVectorizer for TfidfVectorizer on the same dataset and measure whether IDF weighting improves the spam classifier.
Swap BOW for TF-IDF and measure whether IDF weighting improves the same spam classifier on the same data.
Without this:
You would have BOW results without a baseline comparison.
In the previous lesson you built a spam classifier with Bag of Words. The pipeline worked well — but is BOW the best vectorizer for this task? This lesson answers that question by running the same project with one line different: swap CountVectorizer for TfidfVectorizer and compare the results head-to-head.
This is A/B testing at the representation layer — a standard workflow step when iterating on any text classification pipeline.
Why TF-IDF might win on spam:
The key insight is IDF weighting. In a spam dataset, tokens like "FREE", "WINNER", and "URGENT" appear almost exclusively in spam messages — they have low document frequency and therefore high IDF weight. TF-IDF boosts these rare-but-informative tokens relative to common words like "you", "your", "the" that appear in both classes.
Why TF-IDF might not win on spam:
Short SMS messages have few tokens. The IDF boost is meaningful over large corpora (thousands of documents) but may be noisy on 40 messages. Also, spam markers like "FREE" and "CLICK" appear so frequently within spam messages that raw count (BOW) already gives them high weight in the spam class.
This lesson runs: (1) side-by-side classification reports, (2) 5-fold cross-validation for both pipelines, and (3) an ablation with sublinear_tf=True to see whether log-scaling the term frequency adds further signal.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
When TF-IDF beats BOW: rare-token-rich data vs short documents with frequent signal
Why might TF-IDF beat BOW on a spam classifier?
- Swapping CountVectorizer for TfidfVectorizer is one line of code. Always run both with cross_val_score and let the data decide — BOW and TF-IDF are close on small short-text corpora, while TF-IDF wins consistently on large corpora with rare informative tokens.
- 5-fold cross-validation gives a reliable estimate of generalization: mean ± std across folds. A difference smaller than the std is not meaningful — only trust differences larger than 1 standard deviation as indicative.
- sublinear_tf=True replaces TF with 1 + log(TF), preventing a single repeated token from dominating the feature vector. Useful when spammers repeat the same token many times ('FREE FREE FREE') to increase its raw count.
- A/B testing different vectorizers is a standard step when iterating on text classification. The rule: short docs + small dataset → BOW and TF-IDF are close; long docs + large dataset → TF-IDF wins. FeatureUnion can ensemble both if neither clearly dominates.
A/B testing different vectorizers is a standard step in iterating on text classification pipelines. This workflow — fix the model, vary the representation, measure with CV — appears in every production NLP team's process for feature engineering. The same methodology extends to comparing TF-IDF vs sentence embeddings, unigrams vs trigrams, or stemmed vs unstemmed vocabularies.
If you remove it: Stuck with one representation and no head-to-head comparison. You would have no principled way to choose between BOW and TF-IDF — or any two feature representations — beyond intuition.