26 · GRU: a simpler LSTM
GRU merges the LSTM's forget and input gates into a single update gate — fewer parameters, comparable performance on most tasks, drop-in replacement.
GRU merges LSTM's forget and input gates into a single 'update gate' — fewer parameters, comparable performance on most tasks.
Without this:
LSTM is the well-known answer; GRU is the lighter alternative that often matches it.
The LSTM solved the vanishing-gradient problem, but the solution came with four sets of gate weights — roughly four times the parameter count of a vanilla RNN. In 2014, Cho et al. asked: is all that complexity actually necessary?
The Gated Recurrent Unit (GRU) is their answer. GRU strips LSTM down to its essential mechanism. It observes that the forget gate and the input gate are complementary — if you forget a lot, you should be writing a lot of new information, and vice versa. GRU enforces this constraint explicitly by using a single update gate z_t that simultaneously controls how much of the old state to keep and how much of the new candidate to write. The update gate value z_t controls forgetting and writing; 1 - z_t is the complementary coefficient.
GRU also eliminates the separate cell state. There is only one hidden state h_t, and it plays the role of both the cell state and the hidden state in LSTM. This simplification reduces the number of gate weight matrices from 4 (LSTM) to 3 (GRU), cutting parameters by about 25%.
The result is an architecture that is:
- Faster to train — fewer parameters, simpler computation graph.
- More memory-efficient — no separate cell state tensor to store.
- Empirically competitive — on many benchmarks GRU and LSTM are within 1% of each other.
GRU equations: 2 gates (update + reset), no separate cell state
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
GRU vs LSTM: architecture, parameter count, and empirical comparison
PyTorch read-along: nn.GRU as a drop-in for nn.LSTM
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
How does GRU differ from LSTM architecturally?
- GRU has 2 gates (update + reset) and no separate cell state — one hidden state h_t plays both roles.
- GRU has ~25% fewer parameters than LSTM (3 weight matrices vs 4) for the same hidden size and is a direct drop-in replacement via nn.GRU.
- In practice: try GRU first for speed; try LSTM if you suspect very long-range dependencies and have enough data to exploit the extra capacity.
GRU is the go-to in production where latency matters — fewer parameters means faster inference. Bidirectional GRUs power many on-device speech and translation systems. The original neural machine translation seq2seq paper (Cho et al. 2014) used GRUs.
If you remove it: You'd only know LSTM as the gated option, missing a lighter alternative that's standard in resource-constrained deployments and in the seq2seq literature.