25 · Training LSTMs in practice
LSTM training uses the same BPTT machinery as vanilla RNNs — but the cell state's additive update makes long-range gradients survive. Here is what that looks like in numbers.
LSTM training uses the same BPTT machinery as vanilla RNNs — but the cell state's additive update makes long-range gradients survive.
Without this:
Without seeing LSTM training in action you'd assume the architecture is magical; in fact it's just standard backprop with a better gradient flow.
LSTM's architectural novelty is the cell state — but once the network is built, training it uses exactly the same algorithm as a vanilla RNN: Backpropagation Through Time (BPTT). You unroll the computation graph through every timestep, compute the loss at the output, and run loss.backward(). PyTorch does all of that automatically.
What changes is where the gradients flow. In a vanilla RNN every gradient signal must pass through the recurrent weight matrix W_hh and through the tanh derivative (1 - tanh^2) at every single timestep. As soon as those factors are less than 1 (they usually are), the product of 50 such factors rounds to zero. The RNN is blind to events that happened more than ~20 steps ago.
In the LSTM the gradient of the cell state c_t with respect to the previous cell state c_{t-1} is simply f_t — the forget gate value, elementwise. When the forget gate is close to 1 (the network learned to remember), the product of 50 forget-gate values can stay close to 1. This is what Hochreiter and Schmidhuber called the constant error carousel: a gradient path through the cell state that avoids the squashing problem entirely.
But "close to 1" is not guaranteed. The network learns forget-gate values. If the task genuinely calls for forgetting (short-range dependencies), forget gates will be small, and gradients will vanish — intentionally. The LSTM gives the option of long-range memory; it doesn't force it.
Python (in browser)
Gradient magnitude vs steps-back: vanilla RNN vanishes by step 20; LSTM with forget~0.95 survives 100 steps
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The constant error carousel: cell-state gradient is controlled by forget gates, not matrix spectral radii
Gradient clipping is standard for LSTM training — always use clip_grad_norm_ before optimizer.step()
Full LSTM training loop: embed → LSTM → linear head, with gradient clipping
LSTM training tips: forget-gate bias init, hidden size, dropout placement, optimizer
Why do LSTMs avoid vanishing gradients in practice?
- LSTM training is standard BPTT; the gradient advantage comes from the additive cell-state path, not from a different training algorithm.
- Always use gradient clipping (clip_grad_norm_ with max_norm=1.0) when training LSTMs — exploding gradients through the hidden-state path are still possible.
- Initialize forget-gate biases to 1, use dropout between stacked LSTM layers (not within timesteps), and avoid BatchNorm inside LSTMs.
Every LSTM-based model in production uses gradient clipping + forget-gate bias init = 1. These are tiny tricks that turn 'won't train' into 'trains cleanly'. The AWS DeepAR forecasting model, many speech recognition backends, and clinical NLP pipelines all rely on these defaults.
If you remove it: You'd treat LSTM training as a black box, missing the two cheapest interventions (gradient clipping + bias init) that often fix a broken training run before you even need to tune architecture or data.