27 · XGBoost: extreme gradient boosting
XGBoost is gradient boosting on steroids: second-order gradients, smart regularisation, parallel split-finding, and engineering optimisations that make it 10–100× faster than vanilla GB.
XGBoost is gradient boosting on steroids: second-order gradients, smart regularization, parallel split-finding, and engineering optimisations that make it 10-100× faster than vanilla GB.
Without this:
Without XGBoost (or LightGBM/CatBoost) you'd lose to anyone using them — they're the de-facto standard for tabular ML.
Vanilla gradient boosting (sklearn's GradientBoostingClassifier) uses only the first-order gradient (the residual) to guide each tree's split decisions. XGBoost (Chen & Guestrin, 2016) makes four key improvements that explain its dominance:
1. Second-order gradients (Hessian) XGBoost uses both the gradient g (first derivative) and the Hessian h (second derivative) of the loss. The optimal leaf weight becomes:
w* = − Σg / (Σh + λ)
This is a Newton-Raphson step — it converges faster and makes smarter split decisions than first-order methods alone.
2. Regularisation in the objective The XGBoost objective adds an explicit complexity penalty to the loss:
Obj = Σ loss(yi, yi_hat) + γ·T + ½·λ·Σwk² + α·Σ|wk|
where T = number of leaves, λ = L2 on leaf weights, α = L1 on leaf weights. This prevents trees from growing leaves that don't genuinely improve the fit.
3. Column subsampling (colsample_bytree)
Like Random Forest's max_features, XGBoost randomly samples a fraction of columns per tree (or per split), adding diversity and reducing overfitting.
4. Engineering optimisations
- Approximate split finding: pre-sorts data into histogram bins rather than checking every possible split threshold — dramatically faster on large datasets.
- Sparse-aware: efficiently handles missing values and sparse matrices.
- GPU support: tree building can run on GPU with
device="cuda".
Key hyperparameters
| Parameter | Typical | Description |
|---|---|---|
| n_estimators | 200–2000 | Number of trees |
| learning_rate | 0.05–0.1 | Shrinkage per tree |
| max_depth | 3–10 | Tree depth |
| min_child_weight | 1–10 | Min sum of Hessian in a leaf (regularises splits) |
| subsample | 0.6–1.0 | Row subsample per tree |
| colsample_bytree | 0.6–1.0 | Column subsample per tree |
| reg_alpha | 0–1 | L1 regularisation on leaf weights |
| reg_lambda | 1–10 | L2 regularisation on leaf weights |
XGBoost is not available as a Pyodide browser wheel. This code runs locally (`pip install xgboost`) or in Colab. The sklearn-compatible `XGBClassifier` API mirrors `GradientBoostingClassifier` almost exactly — you can swap them with a one-line change.
XGBoost exposes three importance types: `weight` (split count), `gain` (average gain per split — most informative), and `cover` (average sample coverage). `gain` is the recommended default. Run locally with `pip install xgboost matplotlib`.
Python (in browser)
GridSearchCV systematically evaluates every hyperparameter combination using 5-fold cross-validation on the training set — the test set remains untouched. For larger grids, replace `GridSearchCV` with `RandomizedSearchCV` (random sampling) or Optuna (Bayesian optimisation).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
All three libraries share the same underlying gradient boosting algorithm. The differences are in tree-growth strategy, engineering, and how they handle specific data types. In practice, try all three in a hyperparameter search and pick the one with the best cross-validated score.
Compared to sklearn's GradientBoostingClassifier, XGBoost is generally...
- XGBoost uses second-order gradients (Hessian) for smarter, faster split decisions — a Newton-Raphson step vs. gradient descent at the leaf level.
- Regularisation (L1 `reg_alpha`, L2 `reg_lambda`, min leaf penalty `gamma`) is baked into the XGBoost objective — it controls complexity without needing a separate tuning knob.
- XGBoost is NOT available in the Pyodide browser runtime — use sklearn's `GradientBoostingClassifier` in-browser, then swap for XGBoost locally when it matters.
- LightGBM (leaf-wise growth, fastest) and CatBoost (native categoricals, most accurate OOTB) are the main XGBoost alternatives — all three are gradient boosting libraries.
- Always tune XGBoost — untuned XGBoost often loses to a default Random Forest. `learning_rate` and `n_estimators` are the two most impactful knobs.
Won Kaggle's Higgs Boson challenge in 2014; has been on the team of nearly every Kaggle tabular winner since. Production fraud detection (Stripe, Square), recommendations (Airbnb), CTR prediction (ad tech) all use gradient boosting in production.
If you remove it: You'd compete in tabular ML with one hand tied behind your back.