8 · Bias-variance tradeoff
Total expected error = bias² + variance + irreducible noise — learn to diagnose each with learning curves.
Total expected error = bias² + variance + irreducible noise — and you tune model complexity to balance the first two.
Without this:
Without this frame, 'which model should I use?' has no principled answer.
Why does a trained model make errors? There are exactly three sources:
Bias — error from incorrect assumptions built into the model. A linear model trying to fit a cubic curve will always be wrong, no matter how much data you give it. Bias is the systematic component: the model is consistently off in a predictable direction. High bias = underfitting.
Variance — error from the model's sensitivity to the specific training set. A very complex model (e.g., a depth-unlimited decision tree) memorises the training rows — including their noise. Change the training set slightly and the model changes drastically. High variance = overfitting.
Irreducible noise — the floor you cannot beat. Even if you knew the true data-generating function, random noise in the measurements means you can never perfectly predict every point.
The decomposition
For any model and input x, the expected squared error decomposes as:
E[(y − ŷ)²] = Bias(ŷ)² + Var(ŷ) + σ²
Increasing model complexity lowers bias but raises variance. The sweet spot is in the middle.
Diagnosing with training and test error
| Symptom | Cause | Fix | |---|---|---| | High train error + high test error | High bias (underfitting) | More features, more model capacity | | Low train error + high test error | High variance (overfitting) | More data, regularization, simpler model | | Both moderate, gap shrinks with data | Still needs more data | Keep collecting |
The learning curve
Plot train and test error as a function of training set size. Converging to the same (high) error → bias problem. Large gap that persists even with more data → variance problem.
Python (in browser)
High-bias signature: train and val errors converge to the SAME high plateau. Adding more data won't help — the model's form (linear) cannot represent the true quadratic shape.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
High-variance signature: train MSE near zero but a large persistent gap between train and val. More data gradually closes the gap — but regularization (e.g., limiting max_depth) is faster.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The U-shape on the validation curve is the visual signature of the bias-variance tradeoff. The sweet spot is the alpha where val MSE is minimised.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Print this flowchart and stick it near your monitor. Every 'my model is not working' conversation starts here.
Your model has 1% train error and 30% test error. The primary problem is:
- Bias: systematic error from a model that is too simple; variance: sensitivity error from a model that is too complex.
- High-bias signature: train and val errors both high and close together. High-variance signature: train error near 0, large persistent gap with val error.
- More data cures variance but NOT bias — if the model is too simple, more data won't help.
- The validation curve (metric vs hyperparameter) shows the classic U-shape on the val side — its minimum is the sweet spot.
Learning curves are the universal diagnostic plot before choosing a next step. Bias-variance is the conceptual frame for choosing between linear models (high bias) and deep nets / boosted trees (low bias, controlled variance). Regularization is variance reduction.
If you remove it: You'd choose models by accuracy alone and have no recovery path when a model underperforms.