21 · Backpropagation Through Time (BPTT)
BPTT unrolls the RNN into an equivalent deep feedforward network and runs standard backprop — every deep-net trick transfers directly.
BPTT 'unrolls' the RNN into a deep feedforward net (one layer per timestep) and runs standard backprop — that's why RNN training is essentially deep-net training in disguise.
Without this:
Without BPTT, RNN gradients are mysterious; with it, every other deep-net trick (clipping, ReLU, regularization) transfers directly.
In the previous lesson we saw how an RNN processes a sequence forward through time. Now we need to train it — which means computing gradients and updating weights.
The key insight is this: an RNN processing a T-token sequence is mathematically equivalent to a T-layer feedforward network where every layer has identical weights. The hidden state h_1 is computed from h_0 and x_1 (layer 1). The hidden state h_2 is computed from h_1 and x_2 (layer 2). And so on. Each "layer" is just one timestep.
Backpropagation Through Time (BPTT) exploits this equivalence: unroll the RNN, treat it as a deep feedforward network, and run ordinary backprop through the unrolled graph. The gradient flows backwards from the loss at the final step all the way back through timesteps T, T-1, T-2, …, 1.
Because the weights are shared, the gradient for each weight matrix (say W_hh) is the sum of contributions from every timestep:
∂L/∂W_hh = Σ_{t=1}^{T} ∂L/∂h_t · ∂h_t/∂W_hh
This accumulation is the "through time" part. Each timestep votes on the update.
In practice, unrolling through 10,000 timesteps would require 10,000× the memory of a single forward pass. The practical solution is truncated BPTT: backpropagate only through the last K steps (typically K = 20–128), discard gradients further back, and slide a window through the sequence.
Unrolled RNN computation graph — 5-step RNN equals a 5-layer tied-weight feedforward net
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Truncated BPTT — the practical training algorithm for long sequences
If you use truncated BPTT with K=20 and the RNN processes a 100-token sequence, what happens to the gradients for the earliest tokens?
- BPTT unrolls the RNN into a tied-weight feedforward graph and runs standard backprop — no new algorithm needed.
- Gradients accumulate across timesteps: ∂L/∂W_hh = Σ_t contributions from every step.
- Truncated BPTT (K=20–128 steps) is mandatory in practice; it means the model can't learn dependencies longer than K tokens.
Every PyTorch RNN training loop uses BPTT under the hood — the .backward() call on the loss IS BPTT. Truncated BPTT is standard for language modeling; you usually backprop through 35–128 tokens (the chunk size is a hyperparameter you tune).
If you remove it: You'd call optimizer.step() on an RNN without understanding that each weight update is the sum of 35 timesteps' contributions — and you'd have no intuition for why longer backprop windows sometimes help but always cost more memory.