Optimus Agency | Professional Web Development

Ch. 3

\lVert \mathbf{x} \rVert_2 = \sqrt{\textstyle\sum_i x_i^2}

the Euclidean norm

A norm on a vector space V is a function \|cdot\| : V to mathbb{R}_{geq 0} that assigns each vector a non-negative length. Formally, it must satisfy three axioms: absolute homogeneity (\|lambda mathbf{v}\| = |lambda| cdot \|mathbf{v}\|), the triangle inequality (\|mathbf{u} + mathbf{v}\| leq \|mathb

⏱ 2min read◇ 3vizs.✓ 6exercises

📐Math conventionsCheat-sheet: symbols, operators and notation you'll see in this lesson.↗

Before you start

Foundations you'll lean on here

A quick refresher on what this lesson assumes. Tap any card for a plain-language explanation and a tiny worked example.

5 concepts

Big idea

A norm assigns a non-negative length satisfying homogeneity, the triangle inequality, and positive definiteness.

Without this:

Without the norm framework, regularization becomes a mystery knob. Which norm you pick controls sparsity, invariance, and numerical conditioning — and you cannot reason about any of it without this lesson.

A norm on a vector space is a function that assigns each vector a non-negative length. Formally, it must satisfy three axioms: absolute homogeneity (), the triangle inequality (), and positive definiteness ().

The three most important norms in ML form a family. The Euclidean or norm is the straight-line distance you know from school. The Manhattan or norm measures block-by-block travel. The max or norm returns the largest component.

These are unified by the $L_p$ norm: for . As we recover . The unit ball has a distinct shape for each : a diamond for , a circle for , a square for . That shape is not cosmetic — it determines the geometry of regularization.

L¹ → L² → L∞ — animated unit balls

Five phases: build the diamond, swap to the circle, then the square, finally overlay all three. Each ball is the set of vectors of length 1 under that norm.

In ML, norms measure the *size* of parameters or errors. Ridge regression adds to the loss, shrinking weights smoothly toward zero. Lasso regression uses instead; because the unit ball has sharp corners on the axes, the optimum often lands exactly on an axis, producing a sparse weight vector. Switching norms literally changes the geometry of the solution.

Worked example — L2 norm: Let . Compute its length step by step:

Compare

Diamond vs circle — L¹ vs L² unit balls

Slide to scale both unit balls. Notice how the diamond has *corners* on the axes (drives sparsity) while the circle is smooth (shrinks toward the origin uniformly).

L¹ — corners on axes

L² — smooth circle

rball radius1.00

Step-by-step derivation

$L_2$ norm of $(3, 4)$ — Pythagoras lives in every dimension

1/5

1
Definition: square each coordinate, add, square-root.
2
Plug in .
3
Square the components.
4
Sum inside the root.
5
Take the root — the famous 3-4-5 right triangle.

L2 norm of (3, 4)

1/4

Unit balls of $L_1$, $L_2$, $L_\infty$

The set — a diamond, circle, and square respectively. Same vectors, different definitions of length.

Geometrically, this is the length of the vector from the origin to — the famous 3-4-5 right triangle.

Worked example — comparing $L_1$, $L_2$, $L_\infty$: For :

1/2

. Observe — a general inequality that follows from the shape nesting of the unit balls.

Worked example — higher-dimensional L2:

1/3

. In general, for an -dimensional all-ones vector — a useful reference point for normalization.

Three norms of the same vector

For : , , .

A powerful fact: in any *finite-dimensional* vector space, all norms are equivalent in the sense that they induce the same topology — convergence in one norm implies convergence in any other. Practically, this means your choice of norm affects numerical values and geometry, but not qualitative properties like continuity or limits. In infinite dimensions (function spaces), this equivalence breaks, which is why functional analysis becomes far richer.

Exercises

Concept Challenge

Put your understanding to the test. Score + streak + speed all count.

Norms

6 rounds1200 max pts

Active recall

Recap flashcards

1 / 3

Concept

A norm assigns a non-negative length satisfying homogeneity,…

Tap the card to reveal

Checkpoint

Confirm you've got it

3 quick questions. Get 2 right to mark this lesson complete.

Chapters