4 · The vanishing gradient problem
Why deep sigmoid networks failed before ReLU — and the three solutions that unlocked modern deep learning.
In deep nets with sigmoid activations, gradients shrink exponentially as they propagate back — the early layers receive ~0 signal and don't learn.
Without this:
You'd train 50-layer sigmoid nets and wonder why they never converge. The 2012 ReLU paper unlocked deep nets.
Why did it take until 2012 for deep learning to dominate? The math was there from 1986 (backprop). GPUs existed. But training networks deeper than ~5 layers with sigmoid activations was nearly impossible — and now we know exactly why.
The vanishing gradient problem: during backprop, each layer multiplies the gradient by the local activation derivative. Sigmoid's derivative is σ(z)(1-σ(z)), which maxes out at 0.25 (when z=0). Stack 10 layers and the gradient is multiplied by at most 0.25^10 ≈ 9 × 10^{-7} — essentially zero.
The early layers — which learn the most fundamental representations — receive almost no gradient signal. They don't update. The network effectively has only a few working layers near the output.
The fix came in three forms, all still used today:
- ReLU activations — derivative is 1 for positive inputs, so gradients don't shrink
- Careful initialisation (He / Xavier) — keep activation variance stable across layers
- Residual connections (skip connections, ResNet) — provide a gradient bypass around deep paths
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The 3 solutions to vanishing gradient: ReLU, He/Xavier init, residual connections
Gradient Descent — the optimizer that uses these gradients to update weights
Why do deep sigmoid networks fail to train effectively?
- Sigmoid's max derivative is 0.25 — stacking 10 layers shrinks the gradient to ~1e-6, effectively cutting off learning in early layers.
- ReLU's derivative is 1 for active neurons — gradients flow back undiminished, enabling training of 100+ layer networks.
- The three standard fixes: ReLU activations, He/Xavier weight initialisation, and residual (skip) connections.
- Exploding gradient (the opposite problem) is handled by gradient clipping.
Every modern architecture (CNN, ResNet, Transformer) has design choices specifically to fight vanishing/exploding gradients: ReLU/GELU, batch norm, skip connections, layer norm. Understanding the problem makes these design choices obvious.
If you remove it: You'd train deep nets with sigmoid and wonder why they don't converge — and you'd have no framework for understanding why ResNets use skip connections or why Transformers use layer norm.