28 · End-to-end ML project: a full pipeline
Stitch every step into a single reproducible pipeline: load, explore, split, preprocess, train, tune, evaluate, persist. This is the deliverable.
Stitch every step into a single reproducible pipeline: load, explore, split, preprocess, train, tune, evaluate, persist. This is the deliverable.
Without this:
Without an end-to-end mental model, the individual lessons feel disconnected; in production every step matters.
This capstone lesson connects everything you've learned in the Classical Supervised ML track into a single, deployable artefact. The goal is not a new algorithm — it's the shape that every production ML project takes.
The 8 production ML steps
| Step | What you do |
|---|---|
| 1. Load & explore | Understand distributions, missing values, dtypes |
| 2. Train/test split | Reserve a HELD-OUT set — never touch it until step 7 |
| 3. Preprocess | ColumnTransformer + Pipeline to scale/encode features |
| 4. Baseline | Fit the simplest possible model (constant predictor, linear model) |
| 5. Train candidates | Fit multiple algorithms; compare CV scores |
| 6. Tune the winner | GridSearchCV on TRAIN set only — test set still untouched |
| 7. Evaluate honestly | Score the tuned model on the HOLD-OUT test set |
| 8. Persist | joblib.dump(pipeline, path) — save the WHOLE pipeline, not just the model |
Why sklearn Pipeline?
A Pipeline chains preprocessing and model into one object. When you call pipeline.fit(X_train, y_train):
- The
ColumnTransformerfits its scalers and encoders on TRAIN data only. - When you call
pipeline.predict(X_test), the same transformations are applied consistently.
Without a Pipeline, you must manually track and reapply transformations at inference — a common source of data leakage and production bugs.
Data leakage prevention
Fitting a scaler on the full dataset (train + test) before splitting leaks test statistics into the training process. Always: split first, then fit transformers on TRAIN, then transform TEST.
Python (in browser)
Steps 1–3: load the diabetes dataset, split the data, and build a `Pipeline(preprocessor + LinearRegression)`. The key discipline: the test set is created at step 2 and left completely untouched until the final evaluation. The `ColumnTransformer` fits `StandardScaler` on `X_train` only.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Steps 4 and 5: compare four candidate models using 5-fold cross-validation on the training set only. The test set is still untouched. Cross-validation gives an unbiased estimate of generalisation performance for each candidate without ever looking at the holdout.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Step 6: `GridSearchCV` tunes the winning model's hyperparameters using 5-fold CV on the training data only. The `model__` prefix in `param_grid` targets the `model` step inside the `Pipeline`. The result is the best parameter combination found without any contact with the test set.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Steps 7 and 8: the final evaluation uses the test set for the FIRST and ONLY time. `joblib.dump` persists the entire `Pipeline` — both the `StandardScaler` and the `GradientBoostingRegressor` — as a single file. Reloading it gives a ready-to-use inference object that applies the correct transformations automatically.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
A production-ready model is not just a fitted estimator — it is a reproducible, documented artefact that applies identical transformations at inference as at training. This checklist captures the six most common sources of 'model failed in production' bugs.
Why use sklearn's `Pipeline` instead of running preprocessing manually each time?
- The 8-step ML workflow (load → split → preprocess → baseline → candidates → tune → evaluate → persist) is the shape every production ML project takes.
- Always wrap preprocessing and model in a sklearn `Pipeline` — it guarantees identical transformations at training and inference, and allows you to persist the entire workflow as one artefact.
- The test set must be reserved at step 2 and touched ONCE — at step 7. Using it earlier (to pick models, to tune hyperparameters) causes optimistic bias and invalidates the reported performance.
- `joblib.dump(pipeline, path)` persists the complete pipeline (preprocessor + model). Always save and load the FULL pipeline — never the bare estimator.
- The six pre-deployment checks (preprocessing inside pipeline, test set untouched, preprocessor saved, random state pinned, feature order consistent, encoders leak-free) cover the majority of production ML failures.
This is the deliverable. Every production ML project bottoms out at: 'load, preprocess via pipeline, train, evaluate, persist'.
If you remove it: You'd have a stack of Jupyter cells with no deployable artifact.