4 · Polynomial regression & overfitting
Add x², x³, ... as features and the linear model fits curves — but the higher the degree, the more it memorises noise.
Add x², x³, ... as new features and your LINEAR model can fit curves — but the higher the degree, the more it overfits.
Without this:
Without seeing overfitting first-hand, the bias-variance tradeoff stays abstract.
So far our model is restricted to straight lines. The real world is curved. The solution — polynomial feature expansion — is a form of feature engineering: create new features x², x³, ..., xᵈ from the original x, then fit a linear regression on the expanded feature set.
The key insight is that the model is still linear in the expanded features. We're not changing the learning algorithm — we're changing what we feed it. sklearn's PolynomialFeatures(degree=d) automates this transformation.
For a degree-3 expansion on a single feature x:
[x] → [1, x, x², x³]
A linear regression on these four columns fits a cubic polynomial — yet it's still LinearRegression, still minimising MSE, still using OLS under the hood.
The catch: as degree grows, the model gains enormous flexibility. With degree 15 on 50 training points, the model can pass through (or near) every training point — a perfect training MSE. But on new data, those wild oscillations produce terrible predictions. This is overfitting: learning the noise, not the signal.
The bias-variance decomposition formalises the tradeoff:
Total error = Bias² + Variance + Irreducible noise
- Bias = how wrong the model is on average (high for degree 1 on curved data).
- Variance = how much predictions change with different training sets (high for degree 15).
- The sweet spot is a degree where both are acceptably small.
Python (in browser)
`PolynomialFeatures(degree=d)` expands x into [x, x², ..., xᵈ] — the LinearRegression is unchanged. Degree 1 misses the curve; degree 15 wiggles wildly.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Train MSE falls monotonically. Test MSE follows a U-curve — it bottoms out near degree 3 then rises sharply. This U-curve is the bias-variance tradeoff made visible.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
High bias = model too simple for the data. High variance = model too sensitive to training noise. Every ML tuning session is navigating this tradeoff.
If train MSE is 0.01 and test MSE is 2.0, your model is:
- `PolynomialFeatures(degree=d)` expands x to [x, x², ..., xᵈ]. The downstream model is still `LinearRegression` — the non-linearity comes from the features, not the algorithm.
- Degree too low → underfitting (high bias). Degree too high → overfitting (high variance). Use test/validation MSE to find the sweet spot.
- Diagnostic: if train_MSE ≈ test_MSE (both large) → underfitting; if train_MSE << test_MSE → overfitting.
- Total error = Bias² + Variance + Irreducible noise. No model eliminates irreducible noise — lower it by collecting better data, not by adding complexity.
Kernel methods (SVM with RBF) are implicit infinite-degree polynomial regression. Splines and GAMs are constrained polynomial fits. Decision trees achieve non-linearity through partitioning instead of feature expansion.
If you remove it: You can't model any non-linear relationship — and almost all real-world signals are non-linear.