6 · Target encoding and frequency encoding
When a categorical feature is high-cardinality AND your target carries signal, encode by mean target per category — but only after splitting train/test or you'll leak the label.
When a categorical feature is high-cardinality AND your target carries signal, encode by mean target per category — but only after splitting train/test or you'll leak the label.
Without this:
Without target encoding, you either explode dimensionality with OHE on 10,000 ZIP codes or lose the signal entirely by ordinal-encoding a nominal feature.
One-hot encoding on a feature with 10,000 unique ZIP codes produces 10,000 columns — most of which contain almost exclusively zeros. This is the high-cardinality problem: OHE becomes intractable, and ordinal encoding is meaningless for nominal data.
Target (mean) encoding offers a powerful alternative: replace each category with the mean value of the target within that group.
For a regression problem with a city feature and a revenue target:
- Seattle's mean revenue = 82,000 → encode Seattle as 82,000
- Miami's mean revenue = 54,000 → encode Miami as 54,000
This compresses the entire nominal feature into a single, information-rich numeric column — and the signal it captures (how each group relates to the target) is exactly what a model needs.
Three related techniques:
- Mean (target) encoding: replace category with mean target value.
- Target-guided ordinal encoding: rank categories by mean target → assign integer rank 0, 1, 2, … This is a softer version that avoids using raw means as features.
- Frequency encoding: replace category with how often it appears. Useful when popularity itself is predictive (a rare ZIP code may be noisy; a popular one may be a city centre).
The catch: if you compute means using the FULL dataset including the test set, you have data leakage — the model indirectly sees the test labels during training.
Python (in browser)
Mean target encoding: compute the per-group mean of the target, then `.map()` it back. One line of pandas replaces a high-cardinality nominal column with a dense, information-rich numeric feature.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Target-guided ordinal: rank categories by ascending mean target → integer rank 0, 1, 2, … Bounded scale, no raw mean magnitude — more robust for skewed targets.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`df['city'].map(df['city'].value_counts())` — one-liner frequency encoding. Target-agnostic (no leakage risk), captures popularity signal. Works for both supervised and unsupervised tasks.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Smoothed mean = (n × cat_mean + α × global_mean) / (n + α). Rare categories (small n) are pulled toward the global mean; frequent categories keep their group mean. Higher α = stronger regularisation.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The correct target encoding recipe: K-fold out-of-fold encoding during training; full-training-set mean for test. `sklearn.TargetEncoder` (≥ 1.3) implements this automatically.
You target-encode a column with 5 rare categories that each appear exactly once. Without smoothing, what happens at prediction time on a NEW unseen row?
- **Mean (target) encoding** replaces a category with its mean target value — compresses high-cardinality categoricals into a single dense column that directly captures group-level signal.
- **Target-guided ordinal encoding** ranks categories by mean target and assigns integer ranks — a bounded, robust alternative that avoids using raw means as features.
- **Frequency encoding** replaces with category count — target-agnostic (no leakage risk), useful when popularity carries signal, works for unsupervised tasks too.
- **Smoothing** pulls rare category means toward the global mean: (n × cat_mean + α × global_mean) / (n + α). Always smooth when single-observation categories exist.
- Naive target encoding on the full dataset is **data leakage**. Use K-fold out-of-fold encoding during training; `sklearn.preprocessing.TargetEncoder` (≥ 1.3) implements this safely.
Kaggle-winning solutions on high-cardinality categorical data (item IDs, ZIP codes, user IDs) almost always involve target or mean encoding. Click-through rate models in advertising use frequency encoding heavily because ad popularity (how often a creative appears) is itself a strong predictor of future CTR.
If you remove it: Tree models on a 10,000-category column either explode in memory (with OHE) or never split usefully on the raw string labels. Target encoding is the standard bridge between high-cardinality nominal data and any tree or linear model.