Orthonormal Basis

A set of vectors $\{\mathbf{b}_1, \dots, \mathbf{b}_n\}$ is orthonormal if each has unit length ($\|\mathbf{b}_i\| = 1$) and any two are orthogonal ($\langle \mathbf{b}_i, \mathbf{b}_j\rangle = 0$ for $i \neq j$). Compactly, $\langle \mathbf{b}_i, \mathbf{b}_j\rangle = \delta_{ij}$ (the Kronecker delta). An orthonormal basis is a basis that happens to be orthonormal — the best possible coordinate system.

Why 'best possible'? Coordinates in an orthonormal basis are trivially computed: $c_i = \langle \mathbf{v}, \mathbf{b}_i\rangle$. No matrix inversion needed, just dot products. Moreover, lengths and inner products look exactly as in the standard basis: $\langle \mathbf{u}, \mathbf{v}\rangle = \sum_i u_i v_i$ when both are expressed in an ONB. Orthonormality preserves Euclidean geometry.

The Gram–Schmidt process constructs an orthonormal basis from any basis. Given $\{\mathbf{v}_1, \dots, \mathbf{v}_n\}$: set $\mathbf{u}_1 = \mathbf{v}_1$, then iteratively subtract projections onto earlier $\mathbf{u}_i$s: $\mathbf{u}_k = \mathbf{v}_k - \sum_{i<k} \frac{\langle \mathbf{v}_k, \mathbf{u}_i\rangle}{\langle \mathbf{u}_i, \mathbf{u}_i\rangle} \mathbf{u}_i$. Finally normalize each $\mathbf{u}_k$ to unit length. The procedure constructs the QR decomposition $A = QR$ used by many solvers.

Numerically, classical Gram–Schmidt accumulates rounding error badly. Modified Gram–Schmidt reduces this by re-projecting each vector against the running orthonormalized set rather than the originals. For higher stability, Householder reflections or Givens rotations compute QR without ever explicitly forming the intermediate Gram-Schmidt vectors — this is what numpy.linalg.qr does.

Orthonormal bases show up constantly in ML. The Discrete Fourier Transform uses an ONB of complex sinusoids. Wavelet bases provide multi-resolution ONBs of functions. PCA finds an ONB of data-adapted directions, and random orthogonal matrices are used for initialization and low-distortion dimensionality reduction via the Johnson–Lindenstrauss lemma. Whenever you see an orthonormal basis, expect clean math and stable computation.