22 · Bagging: bootstrap aggregating and out-of-bag error
Train many high-variance models on bootstrap samples, then average — variance falls while bias stays, and generalisation improves.
Train many high-variance models on bootstrap samples and average their predictions — variance drops, bias stays put, accuracy rises.
Without this:
Without bagging you'd be stuck with single high-variance models that overfit, or single high-bias models that underfit.
A single deep decision tree is a classic example of a high-variance model: retrain it on a slightly different dataset and the structure can change dramatically. The tree memorises training noise, so test accuracy suffers.
Bagging (Bootstrap Aggregating) is the simplest variance-reduction technique in all of machine learning. The idea is disarmingly straightforward:
- Draw B bootstrap samples — each of size n, sampled with replacement from the original n training rows.
- Train a separate base model on each bootstrap sample independently (trivially parallelisable).
- Aggregate predictions: average for regression; majority vote for classification.
Why does this work?
If the B base models made independent predictions, each with variance σ², their average has variance σ²/B. Real base models aren't independent (they're all trained on overlapping data), so variance reduction is less than 1/B — but still substantial.
Out-of-bag (OOB) error
Because each bootstrap sample contains only ~63.2% of the original rows (on average), the remaining ~36.8% rows were never seen by that particular model. These are the out-of-bag (OOB) samples. For each training row, aggregate predictions from the models that did NOT see it — this gives a free, roughly cross-validation-quality estimate of generalisation error without requiring a held-out validation set.
sklearn API
BaggingClassifier(estimator=..., n_estimators=100, oob_score=True) and BaggingRegressor wrap any sklearn-compatible base estimator.
Python (in browser)
A decision tree grown without a depth limit achieves perfect training accuracy by creating a unique leaf for every training pattern — but this memorisation fails to generalise. The large gap between train and test accuracy is the variance problem bagging is designed to solve.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Wrapping the same deep tree in `BaggingClassifier` dramatically cuts the overfit gap. The OOB score (`oob_score_`) is a free, data-efficient generalisation estimate — built from the ~36.8% of rows each model never saw during training.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The 63.2% figure comes directly from probability theory: each row has a (1 − 1/n)^n chance of being excluded from a bootstrap sample, which converges to 1/e ≈ 36.8% as n grows. The complement — 1 − 1/e ≈ 63.2% — is the expected fraction of unique rows in any bootstrap sample.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The classic ensemble learning curve: accuracy rises steeply with the first few dozen estimators then plateaus. The steep region reflects genuine variance reduction; the plateau means the ensemble has 'used up' the available diversity. There is no catastrophic overfit as n_estimators grows — unlike boosting, you can't add too many base learners in bagging.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The ensemble variance formula ρ·σ² + (1−ρ)/B · σ² shows two levers: more estimators (B↑) and lower correlation (ρ↓). Bagging only pulls the B lever. Random forests pull both — hence the motivation for the random feature subsampling trick in the next lesson.
Bagging reduces variance but not bias. So bagging a high-bias model like a depth-1 decision stump will...
- Bagging trains B base models on bootstrap samples (n rows drawn with replacement) and aggregates predictions — majority vote for classification, average for regression.
- Each bootstrap sample contains ~63.2% unique rows; the other ~36.8% (out-of-bag) give a free, roughly CV-quality generalisation estimate via `oob_score=True`.
- Ensemble variance = ρ·σ² + (1−ρ)/B · σ². Bagging pulls the B lever; lower base-model correlation ρ amplifies the benefit — motivating random forests.
- Bagging works for high-variance base learners (deep trees). It cannot fix high-bias learners — averaging many bad models gives a bad ensemble.
- Test accuracy plateaus around 50–100 estimators — adding more trees beyond that yields diminishing returns at no risk of overfit.
Random forests are bagged decision trees with one extra trick (feature subsampling). The 'bag of models' ensemble pattern shows up in Kaggle solutions, deep learning checkpoint averaging, and dropout (which is conceptually bagging applied to network connections).
If you remove it: You'd lose the simplest variance-reduction technique — and the conceptual bridge to random forests.