Recurrent Neural Networks: RNNs, LSTMs, and GRUs

An RNN processes a sequence $\mathbf{x}_1, \mathbf{x}_2, \dots$ by carrying a hidden state $\mathbf{h}_t = \tanh(W_{xh}\mathbf{x}_t + W_{hh}\mathbf{h}_{t-1} + \mathbf{b})$. Each step folds the new input into a running summary of everything seen so far. Output can be read off $\mathbf{h}_t$ at each step (sequence-to-sequence) or only at the end (classification).

Vanilla RNNs suffer from vanishing and exploding gradients because backpropagation through time multiplies the same recurrent Jacobian repeatedly. LSTMs (Hochreiter & Schmidhuber 1997) fix this with a separate cell state $\mathbf{c}_t$ that flows mostly linearly through time, gated by three sigmoids: input gate, forget gate, and output gate.

The LSTM update is: $\mathbf{f}_t = \sigma(W_f [\mathbf{h}_{t-1}, \mathbf{x}_t])$, $\mathbf{i}_t = \sigma(W_i [\dots])$, $\mathbf{o}_t = \sigma(W_o [\dots])$, $\tilde{\mathbf{c}}_t = \tanh(W_c [\dots])$, $\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t$, $\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)$. The forget gate decides what to keep; the input gate decides what to add.

GRUs (Cho et al. 2014) merge the cell and hidden state and use only two gates — update and reset — getting most of the LSTM benefit with fewer parameters. Both LSTM and GRU were the workhorses of sequence modeling before transformers overtook them around 2017.