Singular Value Decomposition

The singular value decomposition (SVD) factors *any* Matrixⓘ $A \in \mathbb{R}^{m\times n}$ as $A = U\Sigma V^\top$, where $U \in \mathbb{R}^{m\times m}$ and $V \in \mathbb{R}^{n\times n}$ are Orthogonalⓘ and $\Sigma$ is a (rectangular) diagonal matrix with non-negative entries $\sigma_1 \geq \sigma_2 \geq \dots \geq 0$ called the singular valuesⓘ. Unlike eigendecomposition, SVD applies to rectangular and singular matrices — there is no 'diagonalizability' condition to worry about.

Geometrically, every linear map factors as rotate, scale, rotate. $V^\top$ rotates input space to align with the data's principal axes, $\Sigma$ stretches each axis independently (possibly flattening some to zero), and $U$ rotates to output space. This decomposition is why SVD is so powerful: it reveals the intrinsic geometric action of any matrix.

Singular values relate to eigenvalues: the $\sigma_i$ are the square roots of the eigenvalues of $A^\top A$ (or of $AA^\top$). The columns of $V$ are the corresponding eigenvectors of $A^\top A$ (right singular vectors); the columns of $U$ are the eigenvectors of $AA^\top$ (left singular vectors). For symmetric matrices, SVD and eigendecomposition coincide up to sign.

Many matrix diagnostics are read directly off the singular values. The rank of $A$ equals the number of non-zero singular values. The condition number $\kappa(A) = \sigma_1 / \sigma_r$ measures numerical sensitivity — a large condition number means small input errors blow up. The spectral norm is $\sigma_1$, and the Frobenius norm is $\sqrt{\sum \sigma_i^2}$.

Worked example — SVD of a 2×2 matrix: For $A = \begin{pmatrix}3 & 0\\0 & 2\end{pmatrix}$ (already diagonal, positive), the SVD is trivial: $U = I$, $\Sigma = \begin{pmatrix}3 & 0\\0 & 2\end{pmatrix}$, $V = I$. Singular values $\{3, 2\}$; both $U, V$ are the identity. Any diagonal matrix with non-negative entries is its own SVD.

Worked example — singular values of a rectangular matrix: For $A = \begin{pmatrix}1 & 0 & 0\\0 & 2 & 0\end{pmatrix}$ (2×3), compute $AA^\top = \begin{pmatrix}1 & 0\\0 & 4\end{pmatrix}$. Eigenvalues are 1 and 4. So singular values are $\sigma_1 = 2, \sigma_2 = 1$. Rank = 2 (two non-zero singular values), matching that $A$ has full row rank.

SVD is the universal ML tool. The Moore–Penrose pseudo-inverse is $A^+ = V\Sigma^+ U^\top$ — the best-effort inverse for rectangular/singular matrices, used in least-squares regression. PCA on a data matrix $X$ is literally the SVD of (centered) $X$. Latent Semantic Analysis, recommender systems, model compression, and low-rank adaptation all reduce to truncating the SVD to keep only the top singular values.