37 · Element-wise ops, broadcasting, reductions
Vectorise arithmetic across entire arrays without a single Python loop — then master the broadcasting rules that make column-mean centering, z-score normalisation, and softmax one-liners.
**Vectorisation** means expressing a loop over array elements as a single array operation — `a * 2` instead of `for x in a: x * 2`. **Broadcasting** is NumPy's rule for automatically expanding arrays of smaller shapes so they can combine with larger ones: a shape `(3,)` vector can be added to a `(100, 3)` matrix because NumPy 'broadcasts' the vector across all 100 rows. Together these two ideas let you write an entire ML training step — forward pass, loss, gradient — in a handful of one-liners.
Without this:
Without broadcasting, mean-centering a dataset of shape `(1000, 20)` would require explicitly tiling the mean vector into a `(1000, 20)` matrix before subtracting — six lines instead of one. Every normalisation, attention score computation, and batch-norm update relies on broadcasting being implicit and correct.
When you write a + b with two same-shaped NumPy arrays, NumPy adds element a[i] to element b[i] for every i — in C, without a Python loop. This is vectorisation: the loop exists at the C level where it runs at memory-bandwidth speed.
The same applies to unary operations: np.exp(a), np.log(a), np.sqrt(a) each apply the function to every element of a and return a new array of the same shape. These are called ufuncs (universal functions) in NumPy.
Reductions collapse one or more axes: a.sum() returns a scalar; a.sum(axis=0) collapses the row axis and returns one value per column; a.mean(axis=1) collapses the column axis and returns one value per row. The axis parameter is the dimension that disappears after the reduction.
For matrix products, use np.dot(A, B) or the @ operator (A @ B). The @ operator was added to Python 3.5 specifically for matrix multiplication — it calls __matmul__ under the hood.
Python (in browser)
`argmax()` returns the index of the largest element — applied along `axis=1` it gives you the predicted class for each sample in a classification batch (the column with the highest logit score). This is one of the most-called NumPy functions in ML evaluation loops.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Broadcasting aligns shapes **from the right**: `(3, 4)` minus `(4,)` — the shapes match on the trailing axis (both size 4) so NumPy stretches the `(4,)` vector across all 3 rows. Mean-centering a dataset this way is the first step of PCA and z-score normalisation — one subtraction, no loops, no tiling.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The `@` operator is the cleanest way to express a linear transformation. Every fully-connected layer in a neural network is `output = W @ input + bias`. For batched inputs, transpose the weight matrix so dimensions line up: `(batch, in_features) @ (in_features, out_features) → (batch, out_features)`.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Subtracting `x.max()` before `np.exp` is essential — without it, `exp(1000)` overflows to `inf` and the division produces `nan`. The shift does not change the output (it cancels in numerator and denominator) but keeps every exponent in a safe range. This two-line softmax — broadcasting `x.max()` via scalar subtraction, then reducing with `.sum()` — is implemented identically inside PyTorch's `F.softmax`.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Print `.shape` on every intermediate array while learning broadcasting — the rules are mechanical but the silent errors (wrong result, no exception) are nasty. Once you have a mental model of the alignment, broadcasting becomes your most powerful NumPy tool.
What shape does `np.array([1, 2, 3]) + np.array([[10], [20]])` produce?
MML lesson 3 covers matrices as linear transformations. The `@` operator implements exactly this: `W @ x` applies the transformation encoded in `W` to the vector `x`. Broadcasting generalises this: `X @ W.T` applies the same transformation to every row of `X` simultaneously — the same math, batched over samples.
- Element-wise operations (`+`, `*`, `np.exp`, `np.log`, ...) apply in C across the entire array with no Python loop — this is vectorisation.
- Broadcasting aligns shapes from the right; size-1 dimensions stretch to match. The result shape is the element-wise maximum of the aligned shapes.
- Reductions collapse axes: `a.sum(axis=0)` removes axis 0 (rows) leaving per-column totals. The `axis` argument is the axis that disappears.
- `A @ B` is matrix multiplication (`__matmul__`). For neural network layers: `output = W @ input` for a single vector, or `output = X @ W.T` for a batch.
- The numerically stable softmax is `exp(x - x.max()) / exp(x - x.max()).sum()` — the shift cancels algebraically but prevents `exp` overflow in floating point.
Every forward pass in a neural network is a chain of `@` (matrix multiplications), element-wise activations (`np.maximum(0, x)` for ReLU, `softmax` for the output layer), and reductions (`.mean()` for loss). Broadcasting handles the batch dimension transparently: `(batch, features) @ (features, hidden)` gives `(batch, hidden)` with the same one-liner as the single-sample case.
If you remove it: Without vectorised operations and broadcasting, every training step would require four nested Python loops (batch × layer × row × column) — making experimentation effectively impossible at scale. NumPy's vectorisation is the reason you can iterate on a training loop in seconds on a laptop rather than hours.