28 · Encoder-Decoder: the Seq2Seq architecture
Two RNNs in tandem: encoder compresses the input into a context vector; decoder generates output tokens one at a time, initialized with that context.
Two RNNs in tandem: ENCODER compresses the input sequence into a context vector; DECODER generates the output sequence one token at a time, conditioned on that context.
Without this:
Without seq2seq you can't model machine translation, summarization, or any sequence-to-sequence task.
Every RNN architecture we have studied so far maps one sequence to a fixed-length vector (many-to-one) or a same-length sequence (many-to-many). But translation, summarization, and question answering all share a different structure: the input and output sequences have different, unknown lengths. "The cat sat." is three words in English; "Le chat s'est assis." is five words in French.
The sequence-to-sequence (seq2seq) model — introduced by Sutskever, Vinyals, and Le in 2014 — solves this with two stacks talking to each other:
-
Encoder: an LSTM (or GRU) processes the entire input sequence and condenses it into a single vector of fixed size — the context vector (also called the thought vector). This is the encoder's final hidden state.
-
Decoder: a separate LSTM initialized with the context vector. It generates one output token per timestep. Crucially, the decoder is autoregressive: each output token it generates is fed back as the input to the next timestep. The decoder runs until it produces a special
<EOS>(end-of-sequence) token.
The two RNNs have completely separate weight matrices — the encoder learns to summarize input; the decoder learns to expand a summary into output tokens. Only the final hidden state of the encoder is passed to the decoder; after that, the two stacks proceed independently.
Seq2Seq architecture: encoder LSTM → context vector → decoder LSTM autoregressive generation
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Seq2Seq training tricks: teacher forcing, SOS/EOS tokens, scheduled sampling
PyTorch read-along: Seq2Seq class with separate Encoder and Decoder nn.LSTM modules
Why is teacher forcing used during seq2seq training?
- Seq2Seq uses two separate LSTMs: encoder reads the full input and produces a context vector; decoder is initialized with that vector and generates output tokens autoregressively.
- Teacher forcing feeds ground-truth previous tokens during training to prevent error compounding; scheduled sampling gradually shifts to using the model's own predictions for inference-time consistency.
- The bottleneck problem: the entire input is compressed into one fixed-size context vector — this hurts performance on long sequences and motivates attention (lesson 29).
Google's 2016 production translation system was a deep seq2seq with attention. Speech synthesis (Tacotron), summarization, code generation — all are seq2seq variants.
If you remove it: You'd encounter 'encoder', 'decoder', 'context vector', and 'autoregressive decoding' in every NLP paper without understanding what they mean mechanically — concepts that transformer architectures inherited directly from seq2seq.