34 · Masked + cross attention
The decoder block adds two things to the encoder block: MASKED self-attention (no peeking at the future) and CROSS-attention (queries from the decoder, keys/values from the encoder).
The decoder block = MASKED self-attention (a position can't see the future) → Add & Norm → CROSS-attention (Q from decoder, K/V from encoder) → Add & Norm → FFN → Add & Norm.
Without this:
Without masking the decoder would cheat by reading future tokens at training time; without cross-attention it could never look back at the input it is translating.
The encoder block (lesson 33) processes its whole input at once: every position may freely attend to every other position. The decoder block keeps the same skeleton — attention, Add & Norm, FFN, Add & Norm — but adds two new pieces on top of the encoder design.
1. Masked self-attention. The decoder generates tokens one at a time, left to right. At generation time, position t simply does not have access to positions t+1, t+2, … — they haven't been produced yet. During training, though, we feed the whole target sequence at once for parallelism, so we must artificially forbid each position from attending to FUTURE positions. We do this with a causal mask: before the softmax, we add -∞ to every score that looks ahead. After softmax those entries become exactly 0. This is what makes the decoder autoregressive — each prediction depends only on what came before it.
2. Cross-attention. This is the bridge between the two stacks. In the cross-attention sublayer, the queries Q come from the decoder (its current state), while the keys K and values V come from the encoder's output. So the decoder, at each step, queries the entire encoded input and pulls in exactly the information it needs. This is precisely the same information flow as the Chapter 7 seq2seq attention context vector c_t = Σ_i α_{t,i} · h_i — only now expressed as a clean Q/K/V attention over the encoder's hidden states.
So the full decoder block is three sublayers, each wrapped in Add & Norm:
x → Masked Self-Attention → Add & Norm → Cross-Attention(Q=x, K=V=enc_out) → Add & Norm → FFN → Add & Norm
The encoder has two sublayers per block; the decoder has three. That extra cross-attention sublayer is what lets the output stack condition on the input stack.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Dashed green arrows are the residual connections skipping each sublayer.
PyTorch read-along: nn.TransformerDecoderLayer/Decoder, plus nn.Transformer.generate_square_subsequent_mask for the causal mask
Why does the decoder use a causal (look-ahead) mask during training?
- The decoder block = masked self-attention → Add & Norm → cross-attention → Add & Norm → FFN → Add & Norm: three sublayers vs the encoder's two.
- The causal mask adds -inf to future-looking scores before softmax (use np.triu, k=1), making the decoder autoregressive — a position can't see tokens it hasn't generated yet.
- Cross-attention takes Q from the decoder and K/V from the encoder output, which is exactly the Chapter 7 context vector c_t = Σ_i α_{t,i}·h_i in Q/K/V form.
The masked self-attention is the heart of every GPT-style decoder-only LLM; cross-attention powers encoder-decoder models like T5 and the original translation Transformer, and the decoder side of Whisper.
If you remove it: You'd see 'causal mask', 'cross-attention', and 'decoder-only vs encoder-decoder' across every LLM paper without grasping the two small additions that turn an encoder block into a generative decoder.