27 · Bidirectional RNNs
Run two RNNs — forward and backward — and concatenate their hidden states. Each token sees both past and future context, crucial for token-level tasks like NER.
A bidirectional RNN runs TWO RNNs — one forward, one backward — and concatenates their hidden states. Each output position sees both past and future context.
Without this:
Unidirectional RNNs only see the past; for token-level tasks (NER, POS tagging) the next word often gives crucial context.
Consider the sentence: "The bank of the river flooded." A unidirectional RNN processing "bank" has seen only ["The", "bank"] — not enough to decide whether "bank" refers to a financial institution or a riverbank. The unidirectional model won't see "river" until two steps later, by which point "bank" has already been processed and its hidden state passed on.
A bidirectional RNN solves this by running the sequence twice:
- Forward RNN — processes token 1 → 2 → … → T. At position t, it produces
h_forward_tsummarising tokens 1…t. - Backward RNN — processes token T → T-1 → … → 1. At position t, it produces
h_backward_tsummarising tokens T…t.
For each token t, the final representation is the concatenation [h_forward_t; h_backward_t]. The forward hidden state carries context from the LEFT; the backward hidden state carries context from the RIGHT. At "bank", the forward RNN has seen ["The", "bank"] and the backward RNN has seen ["flooded", "river", "the", "of", "bank"] — more than enough to disambiguate.
This doubles the output dimensionality (2 × hidden_size per token) and costs roughly twice the computation — but for token-level tasks where local context on both sides matters, the trade-off is almost always worth it.
Bidirectional RNN: forward pass left-to-right, backward pass right-to-left, concat per token
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
When to use bidirectional: full sequence available + token-level tasks; never for autoregressive generation
PyTorch read-along: bidirectional=True for nn.LSTM and nn.GRU
Why can't you use a bidirectional RNN for next-word prediction?
- A bidirectional RNN runs two separate RNNs (forward + backward) and concatenates their per-token hidden states, giving each position both left and right context.
- Output dimensionality doubles to 2 * hidden_size per token; in PyTorch add bidirectional=True and update downstream layer input sizes.
- Bidirectional models require the full sequence at inference time — never use them for autoregressive generation or online streaming tasks.
BiLSTM-CRF was the state of the art for sequence tagging (NER, POS) from roughly 2015 until BERT arrived in 2018. Production NER pipelines used it for years and many still do in low-resource languages. BERT is the bidirectional transformer that replaced it.
If you remove it: You'd skip directly to BERT without understanding why 'bidirectional' is in its name or why it can't be used for generation — a critical architectural distinction that confuses many practitioners.