30 · Self-attention: every position attends to every other
Self-attention lets every position in a sequence attend to every other position — no recurrence, fully parallel, the engine of transformers.
Self-attention lets every position attend to every other position in the SAME sequence — no recurrence, fully parallel, the engine of transformers.
Without this:
Without self-attention, transformers (the architecture that dominates 2023+) make no sense.
In the previous lesson, attention was defined as: decoder queries encoder hidden states. The query came from one sequence (decoder side); the keys and values came from a different sequence (encoder side). This is cross-attention.
Self-attention removes the encoder-decoder distinction. Every position in a single sequence queries EVERY OTHER position in the same sequence. The query, keys, and values all come from the same source.
Why does this matter? In an RNN, position t can only "see" position t-1 directly. To combine information from positions that are 10 tokens apart, the signal must travel through 10 sequential hidden states, each with its own transformation. Long-range dependencies are hard to capture and require many layers.
In self-attention, every position directly attends to every other position in one step — the distance between positions does not matter. Position t and position t-50 communicate with exactly the same cost as adjacent positions. The mechanism is:
- Project input X of shape
(seq, d)to three matrices: Q (queries), K (keys), V (values), each with a learned linear projection. - Compute scores:
S = Q @ K^T / sqrt(d_k)— shape(seq, seq), one score per pair. - Softmax row-wise: each row is the attention distribution for one output position.
- Aggregate values:
output = softmax(S) @ V— same shape as input.
The / sqrt(d_k) scaling is critical: without it, large d_k causes dot products to grow so large that softmax saturates, making gradients vanish for all non-maximum positions.
Self-attention shapes: X (seq, d) → Q, K, V → scores (seq, seq) → output (seq, d)
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Self-attention (5×5): every row is a probability distribution over all 5 positions — fully parallel, no recurrence
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Why divide by sqrt(d_k): without scaling, large d_k causes softmax saturation and vanishing gradients
PyTorch read-along: nn.MultiheadAttention and manual scaled dot-product self-attention
Self-attention complexity: O(n^2) memory is the bottleneck; full parallelism is why transformers outpace RNNs on GPU
What's the key advantage of self-attention over RNNs for sequence processing?
- Self-attention computes Q = X W_Q, K = X W_K, V = X W_V, then output = softmax(Q K^T / sqrt(d_k)) V — the same input sequence produces all three.
- Dividing by sqrt(d_k) prevents softmax saturation when embedding dimensions are large — the 'scaled' in scaled dot-product attention.
- Self-attention is O(n^2) in memory (the bottleneck for long contexts) but fully parallel on GPU — this is why transformers replaced RNNs as the dominant architecture.
Self-attention is the SINGLE most important operation in modern AI. BERT, GPT, Whisper, Stable Diffusion, AlphaFold — all are built on it. Understanding it is mandatory for understanding any current model.
If you remove it: You'd be able to use transformers but unable to reason about why they work, debug attention patterns, choose appropriate context lengths, or understand the engineering tradeoffs in long-context models like GPT-4 and Gemini.