9 · Putting it together: BOW + n-grams pipeline
Wrap preprocessing + vectorizer + classifier in a sklearn Pipeline for reproducible, production-safe text classification.
Wrap text preprocessing + BOW + n-grams + classifier into a single sklearn Pipeline so train and inference apply the exact same transformations.
Without this:
Production text pipelines drift when training preprocessing diverges from inference preprocessing — silent bugs that are hard to detect.
You now have all the building blocks: tokenization, stopword removal, BOW counts, and n-gram features. In real projects these steps can easily get out of sync — you might fit the vectorizer on training data, save it separately, then forget to apply the same lowercase setting at inference time. The result is a vocabulary mismatch that silently degrades performance.
sklearn's Pipeline solves this by chaining steps into a single object. You call .fit() once on the whole pipeline, and it learns the vocabulary from training data. At inference time .predict() automatically runs the same tokenization, lowercasing, and vectorization that training saw — guaranteed, with no manual bookkeeping.
A text classification Pipeline typically looks like:
Pipeline([
("vec", CountVectorizer(...)), # step 1: raw text -> sparse matrix
("clf", LogisticRegression()) # step 2: sparse matrix -> prediction
])
The double-underscore notation (vec__ngram_range, clf__C) lets GridSearchCV tune hyperparameters of any step by name — the most powerful tuning pattern in classical NLP.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The complete text classification Pipeline checklist
Why use sklearn Pipeline instead of running vectorizer and classifier separately?
- sklearn `Pipeline([('vec', CountVectorizer(...)), ('clf', Classifier())])` is the production-safe way to combine vectorizer and classifier — no data leakage, one artifact to save and load.
- Use `vec__param` and `clf__param` double-underscore notation in GridSearchCV to tune both vectorizer and classifier hyperparameters together.
- Never call `.fit_transform()` on test data — Pipeline handles this automatically by applying only `.transform()` during prediction.
- Persist the entire fitted pipeline with `joblib.dump` — this saves the vocabulary, n-gram settings, and classifier weights as one file, eliminating train/serve skew.
Every text classification system in production uses sklearn Pipeline (or its PyTorch / spaCy / Hugging Face equivalent). The pattern — vectorizer + classifier, tuned with cross-validation, saved as one artifact — is universal across NLP frameworks.
If you remove it: Without Pipeline, training preprocessing diverges from inference preprocessing — 'train/serve skew' — one of the most common and hardest-to-debug production ML bugs.