PCA: Maximum Variance Perspective

Principal Component Analysis (PCA) is the most-used dimensionality reduction technique and a perfect payoff for everything learned in linear algebra. Given data $\mathbf{x}_1, \ldots, \mathbf{x}_N \in \mathbb{R}^D$, PCA seeks an orthonormal basis of directions (principal components) such that projecting onto the first $k$ captures as much variance as possible. The result: a compact, decorrelated representation of high-dimensional data.

Formally, assume the data is centered ($\bar{\mathbf{x}} = \mathbf{0}$) and form the covariance matrix $S = \frac{1}{N} X^\top X \in \mathbb{R}^{D \times D}$. The first principal component $\mathbf{w}_1$ is the unit vector maximizing variance of the projection: By the Rayleigh quotient, the maximum is $\lambda_1$ (the largest eigenvalue of $S$), attained at the corresponding eigenvector. Subsequent components are the remaining eigenvectors in decreasing eigenvalue order, each orthogonal to the previous.

The beauty is that eigenvectors of the covariance matrix are exactly the principal components, and the eigenvalues are exactly the variances captured. Computing PCA reduces to an eigendecomposition of $S$ — or equivalently, an SVD of $X$, which is often more numerically stable. The connection: if $X = U \Sigma V^\top$, then the columns of $V$ are principal components and the squared singular values divided by $N$ are the variances.

The projected data $\mathbf{z}_n = W_k^\top \mathbf{x}_n$ (where $W_k$ stacks the top $k$ eigenvectors) are called scores. Reconstructing gives $\hat{\mathbf{x}}_n = W_k \mathbf{z}_n = W_k W_k^\top \mathbf{x}_n$ — another orthogonal projection, this time onto the top-$k$ principal subspace. The reconstruction error $\sum_n \|\mathbf{x}_n - \hat{\mathbf{x}}_n\|^2$ is minimized; equivalently, PCA is the $k$-rank approximation of $X$ with smallest Frobenius norm (Eckart-Young theorem).

PCA has two equivalent derivations — maximum variance (what we just did) and minimum reconstruction error. They are duals: maximizing the variance kept is equivalent to minimizing the variance discarded. In practice, PCA is used for visualization (first 2-3 components), compression (keep 95% of variance), denoising (drop small-eigenvalue components), and as a preprocessing step before clustering or regression. Its limitations — linear, Gaussian, single-scale — motivate kernel PCA, t-SNE, UMAP, and autoencoders.

Worked example — first PC of 4 points: Data $\{(1,1), (2,2.1), (3,2.9), (4,4.1)\}$ has mean $(\bar x, \bar y) = (2.5, 2.525)$. Centered deviations: $(-1.5, -1.525), (-0.5, -0.425), (0.5, 0.375), (1.5, 1.575)$. The covariance matrix (dividing by $N=4$) is approximately $S = \begin{pmatrix} 1.25 & 1.25 \\ 1.25 & 1.27 \end{pmatrix}$ — nearly rank-1 with both rows equal to $\sim(1.25, 1.26)$. The top eigenvector is $\approx (1, 1)/\sqrt{2}$, the diagonal direction, and $\lambda_1 \approx 2.51$, $\lambda_2 \approx 0.01$ — the first PC captures $\sim 99.6\%$ of variance.