Eigendecomposition and Diagonalization

A square matrix $A$ is diagonalizable if it can be written as $A = PDP^{-1}$, where $D$ is diagonal and $P$ is invertible. The columns of $P$ are eigenvectors of $A$ and the diagonal entries of $D$ are the corresponding eigenvalues. This decomposition expresses the linear map in the *eigenbasis*, where it becomes pure per-axis scaling.

Not every matrix is diagonalizable. A sufficient condition is having $n$ linearly independent eigenvectors, which is automatic when all eigenvalues are distinct. A matrix with repeated eigenvalues may or may not be diagonalizable, depending on whether each eigenvalue has enough eigenvectors (geometric multiplicity = algebraic multiplicity). When it fails, we fall back on Jordan normal form.

Symmetric real matrices are always orthogonally diagonalizable: $A = QDQ^\top$ with $Q$ orthogonal. This is the spectral theorem again — the nicest possible decomposition. For SPD matrices, all eigenvalues are positive, and we can define matrix functions like $A^{1/2} = QD^{1/2}Q^\top$, $e^A = Q e^D Q^\top$, or $\log A = Q \log(D) Q^\top$, simply by applying the function entrywise to the diagonal.

Why diagonalize? Because functions of $A$ become trivial: $A^k = PD^k P^{-1}$ reduces computing a matrix power to raising scalars. Solving linear ODEs $\dot{\mathbf{x}} = A\mathbf{x}$ reduces to decoupled 1D equations. And iterative dynamics like $\mathbf{x}_{k+1} = A\mathbf{x}_k$ have a simple long-term behavior governed by the dominant eigenvalue — if $|\lambda_1| < 1$ the system contracts to zero, if $|\lambda_1| > 1$ it diverges.

In ML, eigendecomposition is the engine of PCA (decomposing the covariance matrix), spectral clustering (decomposing the graph Laplacian), and Gaussian processes (decomposing the kernel matrix). It also underpins stability analysis of learned dynamics (RNNs, diffusion models) — if eigenvalues of the learned Jacobian stray outside the unit circle, training and generation will misbehave in predictable ways.