24 · LSTM: cell state + 3 gates
LSTM adds a cell state that flows through time with minimal modification — three gates decide what to forget, add, and output, fixing the vanishing-gradient barrier.
LSTM adds a 'cell state' that flows through time with minimal modification — gates decide what to forget, what to add, what to output. This is the architecture that broke through the vanishing-gradient barrier.
Without this:
Vanilla RNNs can't learn long-range dependencies; LSTM was the 1997 invention that fixed it.
By the end of chapter 5 you had a precise diagnosis: vanilla RNNs fail on long sequences because gradients vanish as they flow backward through repeated tanh multiplications. Each step multiplies the gradient by a factor that is typically less than 1, and after 50–100 steps the signal becomes numerically indistinguishable from zero. The network simply cannot reach back that far when updating its weights.
The Long Short-Term Memory (LSTM) was Hochreiter and Schmidhuber's 1997 answer. The core insight was surgical: instead of routing information purely through the hidden state h_t — which gets squashed by tanh at every step — add a second, parallel channel called the cell state c_t. The cell state is protected from direct nonlinear squashing; it flows forward through time with only additive updates. When you add rather than multiply, gradients can travel backward without shrinking at every hop.
Three gates — small sigmoid networks whose outputs lie in [0, 1] — control what the cell state remembers and what it exposes:
- Forget gate: which fraction of the old cell state to retain.
- Input gate + candidate: how much new information to write in.
- Output gate: which part of the cell state to expose as the hidden state.
The hidden state h_t still exists and still faces tanh squashing — but now it is a read-out of the cell state rather than the only memory channel. The cell state is the long-term memory; the hidden state is the short-term working memory.
LSTM equations: 4 gates + additive cell state update
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
PyTorch read-along: nn.LSTM interface
What is the FORGET gate's job in an LSTM?
- The LSTM cell state c_t is updated additively: c_t = f_t * c_{t-1} + i_t * g_t — this additive path is what lets gradients survive long sequences.
- Three gates (forget, input, output) are each small sigmoid networks; they output per-dimension control signals in [0, 1].
- LSTM has ~4x more parameters than a vanilla RNN of the same hidden size — a worthwhile trade-off on long sequences, but watch for overfitting on small datasets.
Machine translation 2014–2017 was all LSTM (sequence-to-sequence with attention); speech recognition; time series forecasting. Replaced by transformers post-2017 but still used in low-resource settings.
If you remove it: You'd lose the mechanical intuition for why the cell state solves vanishing gradients — which is also why transformer residual connections work. The additive update principle is a thread that runs through LSTM, GRU, ResNets, and attention.