59 · Capstone: an end-to-end sklearn project
Assemble the estimator API, preprocessing, pipelines, search, and persistence into one end-to-end project — the deliverable shape of a real ML solution.
Everything in this chapter assembles into one end-to-end project: a `ColumnTransformer` for mixed-type preprocessing, wrapped in a `Pipeline` with a classifier, tuned by `GridSearchCV`, evaluated on a held-out set, and persisted with `joblib`. That single saved object — preprocessing and model together — is the deliverable shape of a real ML solution.
Without this:
Knowing each piece in isolation isn't enough; the value is in wiring them into one reproducible artefact you can train once, evaluate honestly, save, and reload to serve predictions — without re-implementing preprocessing by hand.
The capstone: one end-to-end project
This lesson introduces no new concepts. Instead it integrates everything from this chapter — the fit/predict/transform estimator API, scaling and encoding, the ColumnTransformer, the Pipeline, cross-validated search, and joblib persistence — into a single, deployable artefact.
That artefact is the shape every real ML solution ships in:
- Build the data — a mixed-type table of numeric and categorical features with a target.
- Define preprocessing — a
ColumnTransformerrouting numeric columns to aStandardScalerand categorical columns to aOneHotEncoder. - Wrap in a Pipeline — preprocessing + a classifier as one estimator.
- Split honestly —
train_test_split(..., stratify=y, random_state=42). - Tune —
GridSearchCVover a small grid, fit on train. - Evaluate — accuracy + a
classification_reporton the held-out test set. - Persist —
joblib.dumpthe whole fitted pipeline, reload it, and predict on brand-new rows to prove the saved object carries its own preprocessing.
The payoff: the saved file is a self-contained inference engine. Anyone who loads it can call .predict() on raw, un-preprocessed rows and get correct results, because the scaler's learned means and the encoder's learned categories travel inside the pipeline.
Python (in browser)
Steps 1-4: a hand-built 200-row table with two numeric and two categorical columns plus a learnable binary target. A `ColumnTransformer` scales the numerics and one-hot-encodes the categoricals, wrapped with a classifier into one `Pipeline`, then split with `stratify=y`.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Steps 5-6: `GridSearchCV` tunes the pipeline's `model__C` with 5-fold CV on the training data, then we score the winning pipeline on the held-out test set with `accuracy_score` and a full `classification_report` (precision, recall, f1 per class).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Step 7: `joblib.dump` saves the entire fitted pipeline to a file; `joblib.load` brings it back. Predicting on two brand-new raw rows works without any manual preprocessing — the scaler's means and encoder's categories travel inside the saved object. (Mirrors the file-path joblib round-trip proven in `bc-mls-28`.)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The same end-to-end workflow as a runnable `train.py` skeleton: `load_data` → `build_pipeline` → split → `GridSearchCV` → evaluate → `joblib.dump`. Swap `load_data` for a `pd.read_csv` and you have the spine of a real project.
When you `joblib.dump` a fitted Pipeline, what gets saved?
- A real ML deliverable is one object: `ColumnTransformer` → `Pipeline` → tuned by search → evaluated on a held-out set → persisted with `joblib`.
- `joblib.dump(pipe)` saves the WHOLE fitted pipeline — learned scaler means, encoder categories, and the model — so the reloaded object preprocesses and predicts on raw rows by itself.
- Evaluate the tuned pipeline on the held-out test set exactly once, with `accuracy_score` plus a `classification_report` for per-class precision/recall/f1.
This is literally how production tabular ML ships: a single persisted pipeline serving predictions behind an API. The ML-Supervised and NLP tracks reuse this exact scaffold, swapping in stronger models and richer search grids.
If you remove it: Without the end-to-end pattern you'd ship loose scripts that re-implement preprocessing at every prediction site — fragile, leak-prone, and impossible to reproduce reliably.