Orthogonal Complement

Given a subspace $U$ of an inner-product space $V$, its orthogonal complement is the set $U^\perp = \{\mathbf{v} \in V : \langle \mathbf{v}, \mathbf{u}\rangle = 0 \text{ for all } \mathbf{u} \in U\}$ — every vector that is orthogonal to every vector in $U$. This is itself a subspace, and it is the 'leftover' directions that $U$ misses.

The crucial structural fact: $V = U \oplus U^\perp$, meaning every vector $\mathbf{v} \in V$ decomposes *uniquely* as $\mathbf{v} = \mathbf{v}_U + \mathbf{v}_{U^\perp}$ with $\mathbf{v}_U \in U$ and $\mathbf{v}_{U^\perp} \in U^\perp$. This is the orthogonal decomposition and it is the backbone of projections, least squares, and Fourier analysis. Dimensions also add: $\dim U + \dim U^\perp = \dim V$.

A second pleasant property: $(U^\perp)^\perp = U$ — taking the complement twice returns you to the original subspace (in finite dimensions). So $U$ and $U^\perp$ form a perfectly balanced pair, each fully determining the other.

The four fundamental subspaces of a matrix $A \in \mathbb{R}^{m \times n}$ are orthogonal complements in two natural pairings. Inside $\mathbb{R}^n$: the null space $\text{null}(A)$ and the row space $\text{row}(A) = \text{col}(A^\top)$ are orthogonal complements. Inside $\mathbb{R}^m$: the left null space $\text{null}(A^\top)$ and the column space $\text{col}(A)$ are orthogonal complements. This is Strang's fundamental theorem of linear algebra in a single picture.

Applications abound. In regression, the residual vector $\mathbf{y} - X\hat{\mathbf{w}}$ lies in the orthogonal complement of the column space of $X$ — that's *why* the normal equations $X^\top(\mathbf{y} - X\hat{\mathbf{w}}) = 0$ hold. In signal processing, separating a signal into 'useful subspace' plus 'noise subspace' is exactly an orthogonal decomposition. In PCA, the top-$k$ principal directions and the discarded directions are orthogonal complements.