Angles and Orthogonality

From an inner product we can extract the angle between two non-zero vectors via $\cos\theta = \frac{\langle \mathbf{u}, \mathbf{v}\rangle}{\|\mathbf{u}\| \cdot \|\mathbf{v}\|}$. Cauchy–Schwarz guarantees this ratio lies in $[-1, 1]$, so $\theta \in [0, \pi]$ is well-defined. When $\theta = 0$ the vectors are parallel and pointing the same way; when $\theta = \pi$ they are anti-parallel; when $\theta = \pi/2$ they are orthogonal.

Two vectors are orthogonal, written $\mathbf{u} \perp \mathbf{v}$, exactly when $\langle \mathbf{u}, \mathbf{v}\rangle = 0$. Orthogonality is the vector analog of 'independent' or 'unrelated' — orthogonal directions contribute cleanly separate information. Geometrically, two orthogonal vectors meet at a right angle, but algebraically orthogonality extends to any inner-product space, including spaces of functions.

The Pythagorean theorem generalizes to any inner-product space: if $\mathbf{u} \perp \mathbf{v}$, then $\|\mathbf{u} + \mathbf{v}\|^2 = \|\mathbf{u}\|^2 + \|\mathbf{v}\|^2$. This is not a coincidence — the Pythagorean theorem *is* what you get when you expand the squared norm and use orthogonality to kill the cross term. Every 'variance decomposition' in statistics is Pythagoras in disguise.

A matrix $Q$ is orthogonal if $Q^\top Q = I$, i.e., its columns form an orthonormal set (unit length, pairwise perpendicular). Orthogonal matrices preserve inner products: $\langle Q\mathbf{u}, Q\mathbf{v}\rangle = \langle \mathbf{u}, \mathbf{v}\rangle$. Consequently they preserve lengths and angles — they are exactly the rigid motions of space: rotations and reflections. Orthogonal transformations are the best-behaved linear maps, numerically stable and information-preserving.

Worked example — 45° angle: For $\mathbf{u} = (1, 0)$ and $\mathbf{v} = (1, 1)$: $\langle \mathbf{u}, \mathbf{v}\rangle = 1$, $\|\mathbf{u}\| = 1$, $\|\mathbf{v}\| = \sqrt{2}$. So $\cos\theta = 1/\sqrt{2} \approx 0.707$, giving $\theta = 45°$ (or $\pi/4$). Geometrically $(1,1)$ bisects the first quadrant.

Worked example — orthogonality: $\mathbf{u} = (3, 4)$ and $\mathbf{v} = (-4, 3)$: $\langle \mathbf{u}, \mathbf{v}\rangle = 3\cdot(-4) + 4\cdot 3 = -12 + 12 = 0$. Orthogonal, 90° apart. In general, for any $(a,b)$, the vector $(-b, a)$ is perpendicular — a handy 2D trick.

Angles are everywhere in ML. Cosine similarity $\cos\theta$ is the go-to retrieval metric because it ignores vector magnitude. Attention scores in transformers compute $\mathbf{q}^\top \mathbf{k} / \sqrt{d}$ — a scaled dot product, essentially an unnormalized cosine. Orthogonal initialization of weights in RNNs prevents gradient explosion by keeping signal magnitudes stable. The angle concept, once abstract, is the backbone of modern representation learning.