22 · Vanishing & exploding gradients in long sequences
Vanilla RNN gradients shrink or explode exponentially with sequence length — empirically observe why, then see the three escape routes that LSTM, GRU, and transformers use.
Vanilla RNN gradients shrink (or explode) exponentially with sequence length — they can't learn dependencies longer than ~10 tokens. LSTM and GRU (next chapter) fix this.
Without this:
You'd train an RNN on a long sequence and quietly fail to learn anything beyond local patterns — with no error message, just poor accuracy.
In the BPTT lesson we saw that the gradient at timestep 1 flows through T matrix multiplications before it can update any weight. That chain of multiplications is the source of the vanilla RNN's most notorious failure mode.
Recall the chain rule applied to the hidden-state recurrence:
∂h_T / ∂h_1 = ∏_{t=2}^{T} ∂h_t / ∂h_{t-1}
Each factor ∂h_t / ∂h_{t-1} involves the weight matrix W_hh and the tanh derivative (which is at most 1). If the spectral norm (roughly: the largest singular value) of W_hh is less than 1, each multiplication shrinks the gradient. Over T = 100 steps, a shrinkage of 0.9 per step gives 0.9¹⁰⁰ ≈ 2.6×10⁻⁵ — effectively zero. The gradient vanishes.
If instead the spectral norm is greater than 1, each multiplication grows the gradient. Over 100 steps, a growth of 1.1 per step gives 1.1¹⁰⁰ ≈ 13,780. This is an explosion — numerical overflow and wildly diverging training loss.
The tanh activation derivative (max value 1, and ≤ 0.25 in the saturated region) makes vanishing the dominant failure in practice. Gradients for words spoken 30 tokens ago are indistinguishable from zero.
Three escape routes exist:
- Gradient clipping — cap the gradient norm before the optimizer step (prevents explosion but doesn't fix vanishing).
- LSTM / GRU — gates control gradient flow; the LSTM cell state provides a near-constant error gradient path (next chapter).
- Transformers + skip connections — skip connections give every timestep a direct path to the loss; attention removes recurrence entirely.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Mathematical cause of vanishing/exploding gradients and the three escape routes
Why can't a vanilla RNN learn long-range dependencies?
- Gradient magnitude scales as ||W_hh||^T — exponential vanishing when ||W_hh|| < 1, explosion when > 1.
- tanh saturation makes vanishing the default failure mode in practice; vanilla RNNs can't learn dependencies beyond ~10–20 tokens.
- Three escape routes: gradient clipping (stops explosion), LSTM/GRU (gates fix vanishing), transformers (no recurrence at all).
Every modern sequence model uses one of the three escape routes — gradient clipping is universal (clip_grad_norm_ in PyTorch); LSTM/GRU was the dominant answer from 1997–2017; transformers + residual connections are today's answer for almost everything.
If you remove it: You'd wonder why vanilla RNN training silently fails on long texts — and have no mental model for why LSTM was invented in 1997 or why transformers skip recurrence entirely.