20 · RNN vs ANN: handling sequences
MLPs and CNNs fail on variable-length sequences; the RNN fixes both problems by feeding its own output back as input at every timestep.
An RNN feeds its output back as input — at each timestep it sees the current token AND a hidden state summarizing everything before.
Without this:
MLPs and CNNs can't handle variable-length sequences with memory; RNNs were the first architecture that could.
Every architecture you've built so far — MLP, CNN — shares a silent assumption: the input has a fixed shape. An MLP that expects 784 numbers cannot accept 800 numbers without redesigning the weight matrices. A CNN trained on 32×32 images chokes on 64×64 inputs without retraining.
Language doesn't work like that. A tweet has 5 words; a legal document has 50,000. A speech utterance is 0.3 seconds; a lecture is 3,600 seconds. Time-series data — stock prices, EEG signals, sensor streams — arrive in chunks of unpredictable length.
There's a second problem: memory. If you process a sentence word by word with an MLP, each word is processed independently — the network has no way to remember that "not" appeared three words ago when it sees "good". MLPs have no notion of time.
The Recurrent Neural Network was designed to fix exactly these two problems. At its core it is embarrassingly simple: after processing token t, the network saves a hidden state h_t — a vector summarising everything it has seen so far. At the next token, it uses both the new input and that hidden state. That hidden state then replaces itself: h_t feeds into step t+1, which produces h_{t+1}, and so on.
The same weight matrices are reused at every single timestep. This is what makes an RNN independent of sequence length: you could, in principle, feed it 1 million tokens with a fixed parameter count. The recurrence is the memory; the shared weights are the invariance.
The RNN recurrence: h_t = tanh(W_xh @ x_t + W_hh @ h_{t-1} + b)
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Architecture comparison: MLP vs CNN vs RNN
PyTorch read-along: nn.RNN interface
Why does an RNN share weights across all timesteps?
- The RNN recurrence h_t = tanh(W_xh @ x_t + W_hh @ h_{t-1} + b) gives temporal memory with a fixed parameter count.
- Three usage patterns: many-to-one (sentiment), many-to-many (NER/translation), one-to-many (captioning).
- RNNs can't parallelise over the time dimension — that's the fundamental reason transformers replaced them.
Speech recognition, machine translation, language modeling, time-series forecasting — all were RNN-dominated until transformers arrived. RNNs are still used in low-resource edge devices where the small parameter count and low inference latency matter more than parallelism.
If you remove it: You'd jump straight to LSTMs or Transformers without understanding the memory mechanism they're built on. LSTM's cell state, GRU's gating, and transformer's positional encoding all make sense in contrast to the plain RNN recurrence.