Eigenvalues and Eigenvectors

An eigenvector of a square Matrixⓘ $A$ is a non-zero Vectorⓘ $\mathbf{v}$ that $A$ stretches without rotating: $A\mathbf{v} = \lambda \mathbf{v}$ for some scalar $\lambda$, the corresponding eigenvalue. Eigenvectors are the *invariant directions* of the map: every other vector gets bent by $A$, but these lie along axes that $A$ merely scales.

To find them algebraically, rewrite $A\mathbf{v} = \lambda \mathbf{v}$ as $(A - \lambda I)\mathbf{v} = \mathbf{0}$. For a non-trivial $\mathbf{v}$ to exist, $A - \lambda I$ must be singular, i.e., its Determinantⓘ is zero: $\det(A - \lambda I) = 0$. This equation in $\lambda$ is the characteristic polynomial — a polynomial of degree $n$ whose roots are the eigenvalues. For each root $\lambda$, the corresponding eigenvectors span the eigenspace $\ker(A - \lambda I)$.

Eigenvalues can be real or complex, distinct or repeated. A $2\times 2$ rotation matrix has complex eigenvalues $e^{\pm i\theta}$ — it has no real invariant direction other than the origin. A diagonalizable matrix has $n$ linearly independent eigenvectors that form a basis; in that basis, $A$ acts by pure scaling. A defective matrix (like $\begin{pmatrix}1 & 1 \\ 0 & 1\end{pmatrix}$) has fewer independent eigenvectors than its size — it cannot be fully diagonalized and needs Jordan form.

Symmetric real matrices enjoy a remarkable guarantee called the spectral theorem: all eigenvalues are real, eigenvectors from different eigenspaces are orthogonal, and the matrix can be diagonalized by an orthonormal basis: $A = QDQ^\top$ with $Q$ orthogonal and $D$ diagonal. This is the cleanest possible factorization, and it is why covariance matrices, Hessians, and Laplacians are such tractable objects.

Eigenvalues unlock enormous parts of ML. PCA extracts the top eigenvectors of the covariance matrix as principal directions. Spectral clustering uses eigenvectors of the graph Laplacian. Google's PageRank is the dominant eigenvector of a stochastic matrix. Even the stability of a neural-network dynamical system hinges on whether the Jacobian's eigenvalues lie inside the unit disk.