Inner Products

An inner product $\langle \cdot, \cdot \rangle : V \times V \to \mathbb{R}$ is a function that eats two vectors and returns a scalar, satisfying three properties: symmetry ($\langle \mathbf{u}, \mathbf{v}\rangle = \langle \mathbf{v}, \mathbf{u}\rangle$), linearity in each argument, and positive definiteness ($\langle \mathbf{v}, \mathbf{v}\rangle \geq 0$, with equality iff $\mathbf{v} = \mathbf{0}$). The most familiar example is the dot product: $\langle \mathbf{u}, \mathbf{v}\rangle = \mathbf{u}^\top \mathbf{v} = \sum_i u_i v_i$.

Every inner product induces a norm via $\|\mathbf{v}\| = \sqrt{\langle \mathbf{v}, \mathbf{v}\rangle}$. In $\mathbb{R}^n$ with the standard dot product, this gives the familiar Euclidean norm. But we can build *weighted* inner products too: $\langle \mathbf{u}, \mathbf{v}\rangle_A = \mathbf{u}^\top A \mathbf{v}$ for any symmetric positive definite matrix $A$. Different $A$s produce different geometries on the same vector space.

A profound consequence of the axioms is the Cauchy–Schwarz inequality: $|\langle \mathbf{u}, \mathbf{v}\rangle| \leq \|\mathbf{u}\| \cdot \|\mathbf{v}\|$, with equality iff the two vectors are parallel. This single inequality powers countless proofs — the triangle inequality falls out of it, as does the definition of the angle between vectors.

Inner products go far beyond $\mathbb{R}^n$. On the space of continuous functions on $[a, b]$, the integral $\langle f, g\rangle = \int_a^b f(x) g(x) \, dx$ is an inner product. This is the foundation of Fourier analysis: the sine and cosine functions are *orthogonal* under this product, so expressing a signal as a sum of them is the analog of expressing a vector in an orthogonal basis.

Worked example — dot product: $\langle (1, 2, 3), (4, -1, 2) \rangle = 1\cdot 4 + 2\cdot(-1) + 3\cdot 2 = 4 - 2 + 6 = 8$. Positive, so the vectors point in a generally similar direction (angle < 90°).

Worked example — verifying bilinearity: Take $\mathbf{u} = (1, 0)$, $\mathbf{v} = (0, 1)$, $\mathbf{w} = (1, 1)$. Check $\langle \mathbf{u} + \mathbf{v}, \mathbf{w}\rangle = \langle(1,1), (1,1)\rangle = 1 + 1 = 2$, and $\langle \mathbf{u}, \mathbf{w}\rangle + \langle \mathbf{v}, \mathbf{w}\rangle = (1 + 0) + (0 + 1) = 2$ ✓. The inner product distributes over addition.

In ML, inner products are the atom of similarity. Cosine similarity $\frac{\langle \mathbf{u}, \mathbf{v}\rangle}{\|\mathbf{u}\|\|\mathbf{v}\|}$ measures how aligned two vectors are regardless of magnitude — central to word embeddings and retrieval. Kernel methods (like the kernel trick in SVMs) replace the standard dot product with $\kappa(\mathbf{u}, \mathbf{v}) = \langle \phi(\mathbf{u}), \phi(\mathbf{v})\rangle$ where $\phi$ implicitly maps data into a higher-dimensional feature space, never computing $\phi$ directly.