5 · Ridge, Lasso & Elastic Net
Shrink coefficients to fight overfitting — Ridge for stability, Lasso for sparsity, Elastic Net for both.
Regularization shrinks coefficients to fight overfitting — Ridge for stability, Lasso for sparsity, Elastic Net for both.
Without this:
Without regularization, high-degree models and high-dimensional datasets memorise noise instead of learning signal.
The previous lesson ended with a problem: high-degree polynomial models overfit the training data. The root cause is that the coefficients are free to grow as large as needed to pass through every training point. Regularization solves this by adding a penalty term to the loss that discourages large coefficients.
Ridge (L2 regularization): Loss_ridge = MSE + λ · ||w||²
The L2 penalty shrinks all weights smoothly toward zero — but none ever reach exactly zero. Ridge is ideal when you believe all features carry some signal, but want to prevent any single feature from dominating.
Lasso (L1 regularization): Loss_lasso = MSE + λ · ||w||₁
The L1 penalty (sum of absolute values) drives some weights to exactly zero, performing automatic feature selection. The geometric reason: the L1 constraint region is a diamond (in 2D), whose corners lie on the axes — the loss minimum is likely to land on a corner, zeroing out one or more coefficients.
Elastic Net: Loss_en = MSE + λ · [α · ||w||₁ + (1−α) · ||w||²]
A convex combination of L1 and L2. It handles groups of correlated features better than pure Lasso — Lasso arbitrarily picks one from a correlated group; Elastic Net keeps them together with shrunken but non-zero coefficients.
The hyperparameter λ (called alpha in sklearn) controls regularization strength:
- λ = 0 → ordinary linear regression (no penalty).
- λ → ∞ → all coefficients shrink to zero (extreme underfitting).
The right λ is chosen by cross-validation — covered in Chapter 2.
Python (in browser)
Lasso zeros out the noise features (indices 5–19) while keeping the 5 signal features. Ridge shrinks everything but keeps all 20 non-zero.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`RidgeCV` and `LassoCV` perform cross-validation internally to choose the best regularization strength. The `.alpha_` attribute gives you the winner.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Elastic Net with l1_ratio=0.5 produces a hybrid: some exact zeros (from L1) and small but non-zero values for the rest (from L2). Adjust l1_ratio to control the balance.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Default starting point: try Ridge first (stable, fast). Switch to Lasso if you need feature selection. Use Elastic Net when features are correlated and you still want sparsity.
Which regularization produces a coefficient vector with exact zeros?
Probabilistic Modeling — covers the Bayesian interpretation: Ridge = Gaussian prior on weights (MAP estimate), Lasso = Laplace prior. The regularization strength λ corresponds to the inverse prior variance.
- Ridge (L2): adds λ·||w||² — shrinks all coefficients, none to zero. Best for stability and collinear features.
- Lasso (L1): adds λ·||w||₁ — drives some coefficients to exactly zero (feature selection). Unstable with correlated features.
- Elastic Net: combines L1 + L2 via `l1_ratio`. Best when you want sparsity AND want to handle correlated feature groups gracefully.
- Always scale features before regularization — the penalty treats all weights equally regardless of the original units.
- L2 regularization in linear models = weight decay in neural networks. Same formula: loss + λ·||w||².
Every modern ML model has a regularization knob — L1/L2 in linear models, weight decay in PyTorch, max_depth/min_samples in trees, lambda_l1/lambda_l2 in XGBoost. The principle is universal.
If you remove it: You'd have no defense against overfitting except 'more data' — which is rarely available.