Lengths and Distances

Once we have a norm, we automatically get a distance between any two points: $d(\mathbf{u}, \mathbf{v}) = \|\mathbf{u} - \mathbf{v}\|$. The distance inherits all the good behavior of the norm — non-negativity, symmetry, the triangle inequality — so every normed space is also a metric space. And because every inner product induces a norm, every inner-product space automatically comes with a notion of distance.

The Euclidean distance $d_2(\mathbf{u}, \mathbf{v}) = \sqrt{\sum_i (u_i - v_i)^2}$ is the default in most ML contexts — it is the distance you would measure with a ruler. The Manhattan distance $d_1$, Chebyshev distance $d_\infty$, and Mahalanobis distance $d_M(\mathbf{u}, \mathbf{v}) = \sqrt{(\mathbf{u} - \mathbf{v})^\top \Sigma^{-1} (\mathbf{u} - \mathbf{v})}$ (which re-scales by a covariance matrix) are all common alternatives tailored to different data geometries.

A classical identity connects distance to inner product: $\|\mathbf{u} - \mathbf{v}\|^2 = \|\mathbf{u}\|^2 - 2\langle \mathbf{u}, \mathbf{v}\rangle + \|\mathbf{v}\|^2$. This is the law of cosines in vector form, and it is how most similarity-to-distance conversions work under the hood. It also explains why squared Euclidean distance is often preferred computationally: it avoids the square root while preserving ordering.

The closest-point problem — given a query, find the nearest training point — is the heart of $k$-nearest-neighbors classification, retrieval-augmented generation, and clustering. The *choice of distance* changes the answer. For images you may want cosine distance on embeddings; for time series, dynamic time warping; for distributions, the Wasserstein distance. Getting the metric right is half the battle.

Two subtle pitfalls. First, in very high dimensions most pairs of random points are roughly equidistant — the so-called concentration of distances — so naive Euclidean distance loses discriminative power. Second, raw features with different scales (age in years, income in dollars) will be dominated by the larger scale. This is why standardization (subtract mean, divide by standard deviation) or using the Mahalanobis metric is essential before measuring distance on real-world data.