24 · Boosting intro: sequential learning and AdaBoost
Unlike bagging, boosting trains models sequentially — each new learner focuses on the examples the previous ones got wrong, systematically reducing bias.
Boosting trains models SEQUENTIALLY — each new learner focuses on examples the previous ones got wrong.
Without this:
Without boosting you'd never reach the gradient-boosting machines (XGBoost, LightGBM, CatBoost) that dominate tabular ML today.
Bagging and boosting are both ensemble methods, but their philosophies are opposite:
| | Bagging | Boosting | |---|---|---| | Training order | Parallel (independent) | Sequential (dependent) | | Base learner quality | Already good (deep trees) | Weak (stumps, shallow trees) | | Diversity source | Bootstrap resampling | Reweighting by errors | | What it reduces | Variance | Bias | | Risk | Can't overfit with more trees | CAN overfit on noisy data |
The AdaBoost algorithm
AdaBoost (Adaptive Boosting, Freund & Schapire 1995) is the original boosting algorithm:
- Initialise sample weights: wᵢ = 1/n for all i.
- For t = 1, 2, ..., T: a. Train a weak learner hₜ on the WEIGHTED training set. b. Compute weighted error: εₜ = Σ wᵢ · 𝟙[hₜ(xᵢ) ≠ yᵢ] / Σ wᵢ c. Compute learner weight: αₜ = ½ · log((1 − εₜ) / εₜ) d. Update sample weights: wᵢ ← wᵢ · exp(−αₜ · yᵢ · hₜ(xᵢ)) e. Normalise weights so Σ wᵢ = 1.
- Final prediction: sign(Σₜ αₜ · hₜ(x))
Interpretation
- Misclassified examples (yᵢ ≠ hₜ(xᵢ)) get weight multiplied by exp(αₜ) > 1 — HEAVIER.
- Correctly classified examples get weight multiplied by exp(−αₜ) < 1 — LIGHTER.
- Each subsequent stump must pay more attention to the hard examples.
- αₜ > 0 for any learner better than random (εₜ < 0.5); near-perfect learners get very high α.
Python (in browser)
A single depth-1 stump barely beats chance on the moons dataset. AdaBoost's sequential reweighting converts 50 of these weak stumps into a strong ensemble, closing most of the gap to random forest accuracy. This 'weak-to-strong' conversion is AdaBoost's theoretical guarantee (Schapire 1990).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The AdaBoost learning curve differs from bagging's: accuracy rises but can eventually dip on noisy data. Boosting keeps upweighting hard examples — and on noisy datasets, hard examples are often mislabelled, so the algorithm overfits to noise. This is the fundamental risk that bagging doesn't share.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Each depth-1 stump can only draw one straight line. AdaBoost stitches 50 of these together with learned weights αₜ, creating a boundary that looks almost smooth but is actually a staircase mosaic. The sequential focus on hard examples makes the boundary hug the true decision region more tightly than any single stump could.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The bagging/boosting duality is one of the most useful frameworks in classical ML. Whenever a model under- or overfits, ask: is the problem variance (→ bag it) or bias (→ boost it)?
Bagging and boosting both build ensembles, but they differ in...
- Bagging = parallel + independent bootstrap samples → reduces variance. Boosting = sequential + reweights misclassified examples → reduces bias.
- AdaBoost increases sample weights for misclassified examples (×exp(αₜ)) and decreases them for correct ones (×exp(−αₜ)), forcing each subsequent learner to focus on hard cases.
- Boosting CAN overfit on noisy data — unlike bagging, adding more estimators is not always safe. Monitor validation accuracy and use early stopping.
- Boosting's learner weight αₜ = ½·log((1−εₜ)/εₜ) is positive for any learner better than random, and large for near-perfect learners — only accurate stumps get high votes.
- AdaBoost is the conceptual gateway to all modern gradient-boosting machines (XGBoost, LightGBM, CatBoost) — which reframe boosting as gradient descent in function space.
Every gradient-boosting library (XGBoost, LightGBM, CatBoost) is descended from this algorithm. AdaBoost itself is rarely used in production today but it's the conceptual gateway.
If you remove it: You'd lose the sequential-learning paradigm — and gradient boosting's intuition would be opaque.