23 · Random Forest: decorrelated trees and feature importance
Bagging + random feature subsampling at each split decorrelates the trees — the resulting ensemble variance drops further and feature importance becomes trustworthy.
Random Forest = bagged decision trees + random FEATURE selection at each split — decorrelating base learners for even more variance reduction.
Without this:
Without random forests you'd lose the strongest tabular-ML baseline for a decade — and the algorithm that 'just works' without tuning.
Bagging already reduces variance by averaging many base learners. But plain bagging has a ceiling: if all trees are trained on overlapping data and use the same features, their predictions are highly correlated (ρ is large). The ensemble variance formula shows that high correlation limits how much averaging helps.
The random forest trick
At each split — not just at each tree — a random forest considers only a random subset of max_features features as candidates. This single change has a profound effect:
- Trees diverge: with different candidate features at each node, trees structure themselves differently even on similar bootstrap samples.
- Correlation ρ drops: less-correlated trees make the averaging far more effective.
- Each tree is slightly worse than a full-feature tree, but the ensemble wins big.
Typical defaults
| Setting | Classification | Regression |
|---|---|---|
| max_features | "sqrt" (√d) | 1.0 (all, uses bagging only) or "sqrt" |
| n_estimators | 100–500 | 100–500 |
| max_depth | None (full depth) | None |
Feature importance
model.feature_importances_[i] = mean decrease in impurity (MDI) attributable to feature i, averaged across all trees. Because it aggregates across hundreds of trees it's much more stable than single-tree importance — though it still has a slight bias toward high-cardinality numerical features.
Python (in browser)
The progression single tree → bagging → random forest cleanly illustrates the two variance-reduction steps. Bagging reduces variance by averaging; random feature subsampling further reduces inter-tree correlation, squeezing out the final 1-2% improvement.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Random forest feature importance is more stable than single-tree importance because it averages MDI across all 200 trees. The top features (`worst radius`, `worst perimeter`, `worst concave points`) match domain knowledge: cell size and shape irregularity are the strongest indicators of malignancy.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The single tree memorises every training point with jagged step-function predictions. The random forest, averaging 200 trees, produces a much smoother approximation of the underlying signal. Both are piecewise-constant, but averaging hundreds of offset step functions converges toward a smooth curve.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
GridSearchCV over `max_features` confirms that the `sqrt` default is robust — typically within 0.5–1% of the best setting. Setting `max_features=1.0` (no feature subsampling) degrades accuracy because trees become too correlated, which is the plain-bagging result from the first code cell.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Random Forest is the **zero-shot baseline**: high accuracy with zero tuning, interpretable importances, and near-zero risk of catastrophic failure. Only invest the tuning effort in gradient boosting when the RF baseline isn't good enough.
Random Forest's key innovation over plain bagging is...
- Random Forest adds random feature subsampling (`max_features='sqrt'`) at each split on top of bagging — this decorrelates trees, reducing ensemble variance further than plain bagging.
- RF feature importance (MDI) averages impurity reduction across all trees — more stable than single-tree importance but biased toward high-cardinality features.
- The `sqrt` default for `max_features` is usually within 1% of optimal — RF needs almost no tuning to serve as a strong baseline.
- RF is the go-to first model for tabular data: strong defaults, parallelisable, interpretable importances, no scaling needed. Switch to XGBoost when squeezing out the last 1-2% matters.
- For honest feature selection, prefer permutation importance (`sklearn.inspection.permutation_importance`) over MDI — it measures actual accuracy impact on held-out data.
The default tabular ML baseline; sklearn's most-used estimator after LogisticRegression; structured data competitions before XGBoost dominance. Feature importance from RF informs feature selection across countless downstream pipelines.
If you remove it: You'd skip 'the algorithm that just works' — the model you reach for first on any tabular problem.