Maximum Likelihood as Orthogonal Projection

Every algebraic formula has a geometric story. For linear regression, the story is strikingly elegant: least squares is orthogonal projection. The fitted values $\hat{\mathbf{y}} = X\hat{\boldsymbol\beta}$ are the projection of $\mathbf{y}$ onto the column space of $X$ — the subspace of $\mathbb{R}^N$ reachable by linear combinations of the feature columns.

The residual $\mathbf{r} = \mathbf{y} - \hat{\mathbf{y}}$ is orthogonal to every column of $X$: $X^\top \mathbf{r} = \mathbf{0}$. This is exactly the normal equation $X^\top(\mathbf{y} - X\boldsymbol\beta) = \mathbf{0}$. Geometrically, the closest point in the column space to $\mathbf{y}$ is found by dropping a perpendicular; algebraically, the normal equations enforce that perpendicularity. The two views are identical.

The matrix that performs this projection has a name: $P = X(X^\top X)^{-1} X^\top$ is the hat matrix (or projection matrix), because $\hat{\mathbf{y}} = P\mathbf{y}$. It is symmetric ($P = P^\top$), idempotent ($P^2 = P$), and has eigenvalues $0$ or $1$ — hallmarks of a projection. The complementary matrix $I - P$ projects onto the residual space, orthogonal to $\text{col}(X)$. Together they decompose $\mathbb{R}^N$ into signal and residual subspaces.

The trace of $P$ equals the rank of $X$, which equals $D$ for full-rank problems — this is the effective number of parameters. The trace of $I - P$ is $N - D$, the degrees of freedom for residual estimation. The unbiased estimator of noise variance is $\hat\sigma^2 = \|\mathbf{r}\|^2 / (N - D)$, using the dimensionality of the residual space rather than $N$.

This projection viewpoint generalizes beautifully. Ridge regression corresponds to shrinkage of projections along directions of small singular values. Kernel regression projects onto an infinite-dimensional feature space but computes everything via the kernel matrix. Principal Component Regression first projects onto the top principal components, then regresses. Whenever you see a closed-form least-squares solution, there is a projection hiding behind it — and understanding the geometry makes the algebra almost obvious.