20 · Decision tree regression: variance reduction & piecewise prediction
Switch the split criterion from impurity to MSE — the tree now minimises within-child variance and predicts the mean of each leaf's targets.
For regression, trees split to minimise variance (MSE) within child nodes — and predict the MEAN of each leaf's training targets.
Without this:
Without it, you'd miss the regression-version of trees that powers most tabular regression baselines.
The classification tree used impurity (Gini or entropy) to measure how mixed a node's classes are. For regression, there are no classes — there are continuous target values. The natural replacement is variance (equivalently, MSE):
Variance reduction criterion
At each node, compute the mean squared error of the targets in that node:
MSE(t) = (1/n) Σ (yᵢ − ȳ)²
The best split minimises the weighted sum of child MSEs:
Gain = MSE(parent) − [|left|/|parent| · MSE(left) + |right|/|parent| · MSE(right)]
This is equivalent to criterion='squared_error' (sklearn's default for DecisionTreeRegressor).
Leaf prediction
Once the tree is grown, each leaf predicts the mean of its training targets. The resulting function is a piecewise-constant step function: it can only take as many distinct values as there are leaves.
Depth controls the step count
A depth-1 tree has 2 leaves → 2 predicted values. A depth-3 tree has up to 8 leaves → up to 8 predicted values. A depth-10 tree can have up to 1024 leaves — enough to follow individual training noise points closely.
When trees beat linear regression
For non-linear signals (jumps, plateaus, interactions) the tree wins because it makes no linearity assumption. For a truly linear relationship, the tree must approximate a ramp with a staircase — a polynomial or linear model fits it in one shot.
Python (in browser)
The step-function shape is the defining visual of tree regression. Each plateau is one leaf; the height of the plateau is the mean of the training targets that fell into that leaf. With depth=3 and up to 8 leaves the tree captures the broad shape of sin(x) but cannot represent the smooth curvature. Increasing depth adds more steps but risks fitting the noise.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The depth-10 tree has potentially hundreds of leaves and memorises the training noise. The prediction curve oscillates wildly — a perfect training-set fit that will fail on test data. This is the canonical overfit demo for tree regressors. The depth-3 version above is smoother and will generalise better, despite a higher training MSE.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The comparison shows an important bias-variance trade-off: a linear regressor has a strong linearity bias that helps it on linear data but hurts it badly on non-linear signals. A depth-4 tree has no linearity bias and approximates the sine wave with 16 steps. For tabular data with interactions and non-linear relationships, tree models almost always outperform linear baselines.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
A single regression tree is rarely the final model — its value is as a stepping stone to ensembles. But fitting one tree first builds the intuition that makes random forests and gradient boosting click: you're just asking 'what if we combined hundreds of such trees?'
A depth-1 decision tree regressor on 100 points produces a model with how many distinct predicted values?
- Decision tree regression replaces impurity (Gini/entropy) with variance reduction (MSE). At each node, pick the split that minimises the weighted MSE of the two children.
- Leaves predict the mean of their training targets — producing a piecewise-constant step function. A depth-d tree has at most 2^d distinct predicted values.
- Deeper trees overfit noise; shallower trees underfit. The bias-variance trade-off is directly controlled by max_depth.
- Tree regressors cannot extrapolate — predictions outside the training range are clamped to the nearest leaf value. This is a hard limitation for forecasting tasks.
- For non-linear signals, a depth-4 tree already beats linear regression. For linear signals, linear regression wins. Gradient boosting (chapter 8) combines both worlds.
The atomic unit of gradient boosting machines (XGBoost, LightGBM, CatBoost). Standalone regression trees are uncommon, but ensembles of them dominate tabular ML.
If you remove it: Gradient boosting would have nothing to boost.