Orthogonal Projections

An orthogonal projection onto a subspace $U$ is the linear map $\pi_U : V \to V$ that sends each vector $\mathbf{v}$ to its unique best approximation by an element of $U$ — the vector $\mathbf{u}^* \in U$ minimizing $\|\mathbf{v} - \mathbf{u}\|$. By the orthogonal decomposition theorem, this $\mathbf{u}^*$ is exactly the $U$-component of $\mathbf{v}$.

For a 1-D subspace spanned by a unit vector $\mathbf{b}$, the projection is simple: $\pi_{\mathbf{b}}(\mathbf{v}) = \langle \mathbf{v}, \mathbf{b}\rangle \mathbf{b}$. When $\mathbf{b}$ is not unit, divide by its squared norm: $\pi_{\mathbf{b}}(\mathbf{v}) = \frac{\langle \mathbf{v}, \mathbf{b}\rangle}{\langle \mathbf{b}, \mathbf{b}\rangle} \mathbf{b}$. The scalar $\langle \mathbf{v}, \mathbf{b}\rangle / \|\mathbf{b}\|^2$ is the coordinate of the projection along $\mathbf{b}$.

For a general subspace $U = \text{col}(B)$ where $B$'s columns are a basis of $U$, the projection matrix is $P_U = B(B^\top B)^{-1} B^\top$. If the columns of $B$ are orthonormal, this simplifies to $P_U = BB^\top$. Projection matrices are symmetric ($P^\top = P$) and idempotent ($P^2 = P$): projecting twice is the same as projecting once.

The least-squares solution to $A\mathbf{x} = \mathbf{b}$ (when exact solutions don't exist) is exactly $\hat{\mathbf{x}} = (A^\top A)^{-1} A^\top \mathbf{b}$ — the preimage of the projection of $\mathbf{b}$ onto $\text{col}(A)$. Every regression problem, every curve fit, every dimension-reduction step is secretly an orthogonal projection in disguise.

Worked example — project onto a line: Project $\mathbf{v} = (2, 3)$ onto $\text{span}\{(1, 0)\}$. Since $\mathbf{b} = (1, 0)$ is a unit vector, $\pi(\mathbf{v}) = \langle \mathbf{v}, \mathbf{b}\rangle \mathbf{b} = 2 \cdot (1, 0) = (2, 0)$. The residual $(0, 3)$ is perpendicular to the $x$-axis ✓.

Projections also power feature extraction and denoising. PCA projects data onto the top-$k$ principal subspace, discarding noise directions. In image compression, projecting onto a wavelet or DCT basis and keeping the largest coefficients is a lossy projection that the human eye hardly notices. Whenever you approximate something complicated with something simpler, a projection is almost certainly under the hood.