12 · Batch Normalisation
BatchNorm normalises activations to zero-mean unit-variance across the mini-batch — stabilising training and letting you use higher learning rates.
BatchNorm normalizes each layer's activations to zero-mean unit-variance across the mini-batch — stabilizing training and letting you use higher learning rates.
Without this:
Internal covariate shift slows training and forces tiny learning rates; BatchNorm fixes it.
As training progresses, each layer's input distribution shifts because the layers beneath it keep changing. This "internal covariate shift" means each layer is constantly adapting to a moving target — which demands small learning rates and careful initialisation.
Batch Normalisation (Ioffe & Szegedy, 2015) attacks this directly: before the activation function, normalise the pre-activation values across the mini-batch to zero mean and unit variance. Then apply two learnable parameters — scale (γ) and shift (β) — that let the network undo the normalisation if it wants to.
The formula for a single feature dimension across a batch of size B:
mu_B = (1/B) sum(x_i) # batch mean
var_B = (1/B) sum((x_i - mu_B)^2) # batch variance
x_hat = (x_i - mu_B) / sqrt(var_B + e) # normalise (e = 1e-5)
y_i = gamma * x_hat + beta # scale + shift
The net effect: activations entering each layer always have consistent statistics, making the loss landscape smoother and allowing much higher learning rates.
There is one subtle complication: during training, BatchNorm uses the current mini-batch statistics. During inference, there is no batch — we use a running average of the batch statistics accumulated during training. PyTorch stores these as running_mean and running_var. Calling model.eval() switches the layer to use these running stats.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
PyTorch read-along: nn.BatchNorm2d and nn.BatchNorm1d
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Subtle point: why BatchNorm uses different statistics at train vs inference
Why does BatchNorm use different statistics at train vs inference?
- BatchNorm normalises each layer's activations to mean~0, std~1 across the mini-batch — then applies learnable gamma (scale) and beta (shift).
- Training uses live batch statistics; inference uses running averages accumulated during training — ALWAYS call model.eval() before inference.
- BatchNorm allows 10x higher learning rates and acts as regularisation — enabling networks 10x deeper (ResNet era, 2015+).
- For batch sizes <8 or sequence models, use LayerNorm (Transformers) or GroupNorm instead — they don't depend on batch statistics.
Every CNN since 2015 uses BatchNorm. Every Transformer uses LayerNorm (a related but batch-independent variant). Knowing the difference debugs 90% of 'why does my model behave differently at inference' bugs.
If you remove it: Training deep CNNs requires orders-of-magnitude smaller learning rates, converges much more slowly, and is far more sensitive to weight initialisation.