ResNet uses BN after every conv
ResNetBN
Inserted between conv and ReLU. Without it, ResNet-50 doesn't train well.
python
nn.Sequential(nn.Conv2d(...), nn.BatchNorm2d(64), nn.ReLU())Ch. 14 · Deep-Learning Building Blocks
Batch Normalization and Layer Normalization
Deep Learning Building Blocks
Ch. 14
∂W∂L=δa⊤
backpropagation
Batch normalization (Ioffe & Szegedy 2015) normalizes each feature across a mini-batch: hat{x}_{ic} = (x_{ic} - mu_c)/sqrt{sigma_c^2 + epsilon}, then scales and shifts with learnable gamma_c, beta_c. At test time it uses running averages of mu, sigma^2 accumulated during training.
⏱ 1min read✓ 1exercise
Before you start
Foundations you'll lean on here
A quick refresher on what this lesson assumes. Tap any card for a plain-language explanation and a tiny worked example.
★Big idea
BatchNorm normalizes across the batch; LayerNorm across features within an example.
Batch normalization (Ioffe & Szegedy 2015) normalizes each feature across a mini-batch: , then scales and shifts with learnable . At test time it uses running averages of accumulated during training.
BatchNorm dramatically accelerates training of deep nets, allows higher learning rates, and acts as a mild regularizer. The original motivation (reducing 'internal covariate shift') was later questioned — the modern understanding is that BN smooths the loss landscape (Santurkar et al. 2018).
Layer normalization normalizes across features within a single example rather than across the batch. This makes it batch-size-invariant and works on sequences of varying length, which is why it became the default in transformers.
BatchNorm vs LayerNorm by hand
Python (in browser)
Expected: BN zeroes column means; LN zeroes row means.
main.pypython
Loading editor…
Ready — click Run to execute in your browser.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
↻
Other variants: Group normalization groups channels and normalizes per group; instance normalization normalizes each example × channel independently (popular in style transfer); RMSNorm drops the mean centering (used in LLaMA).
In the wild
Where this shows up in production
Each card is a real system, model, or technique that rests on this lesson.
Transformers use LayerNorm everywhere
TransformerLayerNorm
Pre-LN (modern) places LayerNorm before attention/MLP. Stabilizes training of 100+ layer transformers.
RMSNorm in LLaMA / Mistral
LLaMARMSNorm
Drops the mean-subtraction step; only divides by RMS. Slightly faster, comparable quality.
Exercises
Concept Challenge
Put your understanding to the test. Score + streak + speed all count.
Batch Normalization and Layer Normalization
1 rounds200 max pts
Active recall
Recap flashcards
1 / 3
Concept
BatchNorm normalizes across the batch; LayerNorm…
Tap the card to reveal
Confirm you've got it
3 quick questions. Get 2 right to mark this lesson complete.