26 · Gradient boosting: residual fitting and learning rate
Fit each new tree to the RESIDUAL ERRORS of the current ensemble — gradient descent in function space, one weak learner at a time.
Fit each new tree to the RESIDUAL ERRORS of the current ensemble — gradient descent in function space, one weak learner at a time.
Without this:
Without gradient boosting you'd cap at random-forest accuracy; gradient boosting is what unlocks the top of the tabular ML leaderboard.
AdaBoost showed us that sequential learning works: each new learner focuses on what the previous ones got wrong. Gradient Boosting generalises this idea elegantly — instead of reweighting samples, it fits each new tree directly to the residual errors (negative gradients of the loss function) of the current ensemble.
Think of it as gradient descent in function space. In ordinary gradient descent you update a vector of weights. In gradient boosting you update a function — the ensemble itself — by adding a new tree that points in the direction of steepest descent of the loss.
The algorithm (regression, squared loss)
- Initialise: F₀(x) = mean(y) — the simplest constant predictor.
- For t = 1, 2, ..., T: a. Compute negative gradients (= residuals for squared loss): rᵢ = yᵢ − Fₜ₋₁(xᵢ) b. Fit a shallow tree hₜ to predict the residuals rᵢ. c. Update: Fₜ(x) = Fₜ₋₁(x) + η · hₜ(x) where η is the learning rate (shrinkage).
- Final prediction: Fₜ(x).
Key hyperparameters
| Parameter | Typical range | Effect |
|---|---|---|
| n_estimators | 100–2000 | More trees → better fit, risk of overfit |
| learning_rate (η) | 0.01–0.3 | Smaller η → needs more trees, but generalises better |
| max_depth | 3–8 | Shallower than RF — ensemble does the heavy lifting |
The learning_rate / n_estimators tradeoff: halving η and doubling n_estimators typically gives the same or better test error, at double the training cost. The "right" η is usually found by grid search or Optuna.
sklearn API: GradientBoostingRegressor and GradientBoostingClassifier. Both expose staged_predict — an iterator that yields predictions at each boosting stage, allowing you to monitor the learning curve without retraining.
Python (in browser)
Each panel shows the cumulative ensemble prediction after 0, 1, 2, and 3 boosting iterations. The flat baseline (mean of y) is progressively refined as successive depth-3 trees fit the residuals. Notice the RMSE dropping at each step — gradient boosting in its purest form.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`GradientBoostingRegressor(n_estimators=200, learning_rate=0.05, max_depth=3)` typically outperforms both a single deep tree and a Random Forest on smooth regression problems. The key insight: GB simultaneously controls bias (via more estimators) and variance (via shallow trees + small learning rate).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Smaller learning rates need more trees to converge, but they generalise better — especially at large `n_estimators`. A large η (0.5) can overshoot and actually WORSEN with more trees due to variance creep. The sweet spot is usually η ∈ [0.05, 0.1] with 200–500 trees.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`staged_predict` yields test-set predictions at every boosting stage. Plotting test accuracy vs stage number shows you exactly where adding more trees stops helping — this is the gradient boosting equivalent of 'early stopping'. The optimal `n_estimators` is the stage with peak test accuracy.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Gradient boosting is not restricted to squared error. Any differentiable loss yields a valid algorithm — the tree at each step fits the negative gradient of that specific loss. This makes GB applicable to quantile regression, robust regression, and classification under a single unified framework.
In gradient boosting regression with squared loss, what does each new tree predict?
- Gradient boosting fits each new tree to the RESIDUALS (negative gradients of the loss) of the current ensemble — it is gradient descent in function space.
- The learning rate η shrinks each tree's contribution; smaller η + more trees typically beats larger η + fewer trees — at the cost of training time.
- Trees in GB should be SHALLOW (depth 3–8) — individual trees are intentionally weak; the ensemble does the expressive work.
- `staged_predict` lets you monitor the learning curve and pick the optimal `n_estimators` without retraining — the GB equivalent of early stopping.
- GB is NOT parallelisable across trees (each depends on the previous) — it is slower than RF for the same tree count. XGBoost and LightGBM compensate with engineering optimisations on individual tree construction.
The dominant algorithm for tabular ML — Kaggle competitions, fraud detection, click-through rate prediction, demand forecasting all use gradient boosting. XGBoost, LightGBM, and CatBoost are gradient boosting implementations.
If you remove it: You'd hit the random forest ceiling and miss the algorithm category that wins 80% of tabular ML competitions.