Regularization: L1, L2, and Elastic Net

Models that fit training data perfectly often fail in production. The cause is overfitting: when a model has more capacity than the data warrants, it memorizes noise instead of pattern. Regularization is the umbrella term for all the ways we add a *prior preference for simpler models* to the loss, fighting overfitting without throwing away data. It is the single most important practical technique in supervised learning, and almost all of it is built from two norms — $L_1$ and $L_2$.

Add a penalty term to the loss: The hyperparameter $\lambda \geq 0$ trades off fitting the data against keeping $\boldsymbol\theta$ small. Ridge regression uses $\Omega = \|\boldsymbol\theta\|_2^2$ (sum of squares); Lasso uses $\Omega = \|\boldsymbol\theta\|_1$ (sum of absolute values); Elastic Net mixes both. The choice of norm changes the geometry of the optimum dramatically.

Ridge has a closed-form solution: $\hat{\boldsymbol\theta}_{\text{ridge}} = (\mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I})^{-1} \mathbf{X}^\top \mathbf{y}$. The $\lambda \mathbf{I}$ term guarantees the matrix is invertible *even when* $\mathbf{X}^\top \mathbf{X}$ is rank-deficient — this is why ridge is the workhorse for ill-conditioned regressions. Lasso has no closed form (the absolute value is non-differentiable at zero), but coordinate descent and proximal gradient methods solve it efficiently and the resulting solutions are sparse — many coordinates are exactly zero.

Why does $L_1$ produce sparsity but $L_2$ does not? Picture the level sets. The unconstrained loss has elliptical contours; the regularizer's level sets are balls in the chosen norm. The optimum is where the loss contour first touches the ball as it grows. The $L_2$ ball is a sphere — contact is generic and the touchpoint has all coordinates non-zero. The $L_1$ ball is a *cross-polytope* with sharp corners on the coordinate axes. Contours overwhelmingly first meet the $L_1$ ball at a corner, where some coordinates are exactly zero. Geometry forces sparsity.

From the Bayesian view, regularization is a prior on parameters and the regularized loss is the negative log-posterior. $L_2$ corresponds to a Gaussian prior $\boldsymbol\theta \sim \mathcal{N}(\mathbf{0}, \sigma^2 \mathbf{I})$; $L_1$ corresponds to a Laplace prior. Maximum a posteriori (MAP) inference under these priors gives ridge and lasso exactly. Tuning $\lambda$ is tuning the prior strength.

Worked example — ridge shrinks coefficients: Tiny dataset $\mathbf{X}$ has columns $(1, 1, 1)^\top$ and $(1, 2, 3)^\top$, target $\mathbf{y} = (2, 3, 5)^\top$. The OLS solution is $\hat{\boldsymbol\theta} = (0.333, 1.5)^\top$. With $\lambda = 1$, ridge gives $\approx (0.591, 1.154)^\top$. Both coefficients shrank — toward zero but not all the way.

Worked example — soft-thresholding is the Lasso update: Single-feature lasso reduces to $\hat\theta = S_\lambda(\theta_{\text{ols}})$ where $S_\lambda(z) = \mathrm{sign}(z) \cdot \max(|z| - \lambda, 0)$. With $\theta_{\text{ols}} = 0.4, \lambda = 0.5$: $\hat\theta = 0$. With $\lambda = 0.3$: $\hat\theta = 0.1$. Sparsity in action.