25 · AdaBoost: step-by-step implementation walkthrough
Walk through every step of AdaBoost on a 6-point dataset — weight updates, learner weights, and a second iteration — then verify against sklearn.
Walk through every step of AdaBoost on a 5-point dataset — and the math stops being mysterious.
Without this:
Without seeing AdaBoost step-by-step, the weight updates and learner weights feel arbitrary.
The previous lesson introduced AdaBoost conceptually. This lesson makes it concrete by tracing the algorithm through two complete iterations on a tiny 6-point binary dataset. After following the numbers, the update rule no longer feels like magic — it's just exponential reweighting guided by each learner's accuracy.
The 6-step algorithm (recap)
Given training data (xᵢ, yᵢ) with yᵢ ∈ {−1, +1} and T weak learners:
- Initialise: wᵢ = 1/n for all i.
- For t = 1 to T: a. Train hₜ on the weighted dataset. b. Compute εₜ = Σᵢ wᵢ · 𝟙[hₜ(xᵢ) ≠ yᵢ] c. Compute αₜ = 0.5 · log((1 − εₜ) / εₜ) d. Update: wᵢ ← wᵢ · exp(−αₜ · yᵢ · hₜ(xᵢ)) e. Normalise: wᵢ ← wᵢ / Σⱼ wⱼ
- Final: H(x) = sign(Σₜ αₜ · hₜ(x))
Note: in sklearn's SAMME algorithm for classification, the label encoding uses {0, 1} rather than {−1, +1}, but the weight-update logic is equivalent.
Why exponential updates?
The update rule comes from minimising an exponential loss:
L = Σᵢ exp(−yᵢ · F(xᵢ))
where F(xᵢ) = Σₜ αₜ · hₜ(xᵢ) is the cumulative ensemble score. At each step, the greedy minimiser of this loss picks hₜ and αₜ exactly as described above. This exponential loss lens is the bridge between AdaBoost and gradient boosting — in gradient boosting, you can swap in any differentiable loss.
Python (in browser)
AdaBoost iteration 1 in full detail. After fitting the first stump, misclassified points receive weights larger than 1/n while correctly classified points shrink below 1/n. The new weight distribution tells the second stump exactly which examples matter most.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Iteration 2 trains on the reweighted data — the second stump focuses on the examples the first stump got wrong. The weighted combination of both stumps often classifies all 6 training points correctly, demonstrating how sequential reweighting turns two weak learners into a perfect (on training data) strong learner.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`ada.estimator_weights_` exposes sklearn's internal αₜ values. They should roughly match the hand-calculated alphas from the previous cells — any small discrepancy is due to sklearn's label-encoding convention ({0,1} vs {−1,+1}) and internal normalisation. The stump split features and thresholds confirm the same trees were learned.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The exponential-loss derivation is why AdaBoost is the conceptual ancestor of all gradient-boosting machines: replace the exponential loss with log-loss and you get logistic-regression boosting; replace it with squared error and you get gradient-boosted regression trees. The family tree is one elegant idea iterated.
After one AdaBoost iteration, mislabelled examples have weights that are __ compared to correctly-labelled ones.
- AdaBoost's weight update wᵢ ← wᵢ · exp(−αₜ · yᵢ · hₜ(xᵢ)) multiplies misclassified-point weights by exp(+αₜ) > 1 and correct-point weights by exp(−αₜ) < 1 — precisely focusing the next stump on hard cases.
- Learner weight αₜ = 0.5·log((1−εₜ)/εₜ) is large when εₜ is small (accurate stump) and approaches 0 as εₜ → 0.5 (random stump). Only stumps better than chance contribute.
- AdaBoost is gradient descent on exponential loss L = Σᵢ exp(−yᵢ·F(xᵢ)) — the conceptual bridge to gradient boosting, where you swap in any differentiable loss.
- Use `algorithm='SAMME'` for hard-label base learners without `predict_proba`; use `algorithm='SAMME.R'` (default) for probability-based updates when the base estimator supports it.
- Two iterations of AdaBoost on a 6-point dataset can correctly classify all training examples — each stump corrects a different subset of errors through reweighting.
The conceptual ancestor of all modern gradient boosting machines. Understanding AdaBoost is the prerequisite for understanding XGBoost's internals.
If you remove it: You'd treat XGBoost as a black box.