9 · Exploding gradient + gradient clipping
The flip side of vanishing — gradients that grow exponentially — and the one-line fix that keeps RNNs and Transformers stable.
The opposite of vanishing — exploding gradients — happens when weights are too large; clip the gradient norm before each update.
Without this:
Without clipping, RNNs and transformers training on long sequences regularly blow up to NaN.
In lesson 4 we studied vanishing gradients: signals shrink to ~0 as they backpropagate through sigmoid layers. The opposite failure mode exists too — exploding gradients — and it's just as common in deep sequence models.
When network weights are initialised too large, each layer multiplies the gradient rather than shrinking it. Through 20 layers, a gradient that starts at 1 can reach 10^{20} — sending weights to NaN and crashing training within a few hundred steps.
Exploding gradients are especially common in:
- RNNs and LSTMs — the same weight matrix
Wis multiplied at every time step; for long sequences this is equivalent to raisingWto the T-th power - Transformers without proper initialisation or warmup
- Very deep feedforward nets with poorly scaled weights
The standard fix is gradient clipping by norm: if the global gradient norm exceeds a threshold, scale all gradients down proportionally so the norm equals the threshold. This preserves the direction of the update while bounding its magnitude.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
PyTorch read-along — clip_grad_norm_ in a training loop
Which gradient clipping method preserves the direction of the update?
- Exploding gradients are the mirror image of vanishing: large weights amplify gradients exponentially layer by layer, sending weights to NaN within a few hundred training steps.
- Gradient clipping by norm: `if ||g|| > max_norm: g = g * max_norm / ||g||`. Scales all gradients uniformly, preserving the update direction.
- Clipping by value changes direction; clipping by norm preserves direction — prefer norm clipping in practice.
- In PyTorch: call `torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)` after `loss.backward()` and before `optimizer.step()`. It is standard in every RNN/LSTM training loop.
Every RNN/LSTM training loop includes `clip_grad_norm_`. Transformer training often uses it too. Without it, sequence models routinely diverge.
If you remove it: You'd train RNNs that explode to NaN after 100–200 steps and have no systematic way to stabilise them — the fix is literally one line of code.