56 · Transformers & preprocessing
Models need numeric, comparably-scaled inputs and encoded categories — transformers learn the statistics with `fit` and apply them with `transform`.
Most models need numeric, comparably-scaled inputs and encoded categorical features. Transformers handle this with the same two-step rhythm: `fit` learns the statistics (means, standard deviations, the set of categories) and `transform` applies them — and the order matters enormously for avoiding leakage.
Without this:
Feed raw, unscaled, mixed-type data to most models and they break or perform terribly: distance-based and gradient-based methods are dominated by whichever feature happens to have the largest numbers, and they cannot ingest text categories at all.
Why raw data is rarely model-ready
A model only sees numbers, and it treats those numbers literally. Two problems follow:
- Scale. If
incomeranges 0–100,000 andageranges 18–90, a distance- or gradient-based model will be overwhelmingly driven byincomepurely because its numbers are larger — not because it matters more. Scaling puts every feature on a comparable footing. - Categoricals. A column like
city = ["Madrid", "Lima", "Bogotá"]is text. The model cannot multiply text by a weight. We must encode categories into numbers — and crucially, do so without inventing a false ordering.
Transformers solve both. They follow the same contract you met last lesson:
fit(X)— learn the statistics from data (the column means and standard deviations for a scaler; the set of distinct categories for an encoder). The learned values land in trailing-underscore attributes likemean_,scale_,categories_.transform(X)— apply those learned statistics to produce a new feature matrix.
The same scaler, once fit, can transform any data of the same shape — which is exactly what lets you apply your training-set statistics to brand-new test or production rows.
Python (in browser)
`StandardScaler` learns each column's mean (`mean_`) and standard deviation (`scale_`) during `fit`, then `transform` outputs z-scores: mean ≈ 0, std ≈ 1. `MinMaxScaler` is the alternative that rescales to [0, 1] instead.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The scaler's `mean_` matches the training columns' means exactly — proof it learned only from train. The test set, scaled with those frozen train statistics, ends up with means slightly off zero, which is correct and honest.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`OneHotEncoder` creates one binary column per category and invents no ordering — the safe default for nominal data. `OrdinalEncoder` maps to integers and is only appropriate when categories are genuinely ordered. `handle_unknown="ignore"` keeps unseen categories from crashing at predict time.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`ColumnTransformer` applies `StandardScaler` to the numeric columns and `OneHotEncoder` to the categorical column, then concatenates the results. Two numeric + three one-hot columns produce a 5-column output — all in a single `fit_transform`.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Why must you fit a scaler on the training set only, and not on the full dataset?
- `StandardScaler` centers features to mean 0 / std 1; `MinMaxScaler` rescales to [0, 1]. Both learn statistics in `fit` and apply them in `transform`.
- Always fit on train only, then transform both train and test. Fitting on test/full data leaks the test distribution and inflates scores.
- Encode categories with `OneHotEncoder` (no false ordering) or `OrdinalEncoder` (only when genuinely ordered); `ColumnTransformer` routes each column group to its own transformer.
Almost every real tabular ML pipeline begins with scaling numeric features and encoding categoricals via a `ColumnTransformer`. Distance-based (KNN, SVM) and gradient-based (linear, neural) models are especially sensitive to unscaled inputs.
If you remove it: Skip preprocessing and your features arrive on incompatible scales (one dominates the rest) and your categorical columns can't be consumed at all — many models simply error out or collapse to near-random performance.