29 · Attention: the decoder looks at every encoder state
Attention lets the decoder query all encoder hidden states at every output step — weighted by relevance — fixing the context-vector bottleneck.
Attention lets the decoder look at ALL encoder hidden states at every output step — weighted by relevance to what's being generated.
Without this:
Plain seq2seq compresses an entire input into a single context vector; attention fixes this by letting the decoder query the full input history.
In the last lesson we built seq2seq and saw its central weakness: the entire meaning of a sentence — however long — must fit in a single context vector of fixed size. For short sentences this works surprisingly well. For long sentences (20+ tokens), performance degrades sharply.
Think about what a human translator does. When translating "The cat sat on the mat" to French and generating the word "chat" (cat), you are not trying to remember the whole English sentence — you're looking specifically at "cat." When generating "sur" (on), you look at "on." Attention gives the neural network the same capability: at each output step, look back at ALL the encoder's hidden states and decide which ones are most relevant right now.
Formally, at decoder step t with current hidden state s_t:
- Alignment scores: for each encoder position
i, compute a scalar scoree_{t,i} = score(s_t, h_i)measuring how much positionimatters for this output step. - Attention weights: softmax over scores:
α_{t,i} = exp(e_{t,i}) / Σ_j exp(e_{t,j}). These sum to 1 and form a soft distribution over input positions. - Context vector: weighted sum of encoder hidden states:
c_t = Σ_i α_{t,i} · h_i. - Decoder input: concatenate
c_twith the decoder's current state and feed into the next LSTM step.
The key insight: c_t is different at EVERY output step. The decoder gets a fresh, task-specific summary of the input at each timestep — not one fixed vector for the whole output sequence.
Attention recipe: alignment scores → softmax → weighted sum of encoder states → context vector
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Hover a row (target token) to highlight its attention distribution. Switch heads above.
Python (in browser)
Attention heatmap for a 5-token source, 4-token target pair — brighter cells indicate higher attention weight
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Bahdanau vs Luong attention: scoring functions, where context is injected, parameter cost
Attention as differentiable soft lookup: query, keys, values — the abstraction that leads to transformers
What does the attention weight α_{t,i} represent?
- Attention computes a dynamic context vector c_t at each decoder step as a softmax-weighted sum of ALL encoder hidden states — not one fixed vector for the whole output.
- The three score functions are dot-product, bilinear (Luong), and MLP (Bahdanau). Dot-product attention generalizes directly to transformer self-attention.
- Attention is a differentiable soft lookup: query, keys, values — the same abstraction that powers BERT, GPT, and every modern transformer.
Attention is THE central idea of transformers. The same softmax(QK^T)V formula runs in BERT, GPT, T5, Stable Diffusion, AlphaFold — all of them.
If you remove it: You'd use transformers as black boxes without understanding why query-key-value is the right abstraction — a gap that makes debugging, fine-tuning, and architecture choices opaque.