GPT, LLaMA, Claude — every modern LLM
GPTLLaMATransformer
Decoder-only transformers stack 32-100 layers of masked multi-head attention + MLP. Every token in every layer is a softmax-weighted average of prior tokens — that's the whole architecture.
Ch. 13 · Deep Learning Foundations
Attention: Q, K, V and Scaled Dot-Product
Deep Learning Foundations
Ch. 13
a=σ(Wx+b)
a neural layer
Attention is the operation that powers every modern transformer — the architecture behind GPT, Claude, BERT, and essentially every state-of-the-art language and vision model. At its mathematical core, attention is astonishingly simple: a soft-weighted lookup in a learned key-value store. Once you se
⏱ 2min read◇ 2vizs.✓ 5exercises
Before you start
Foundations you'll lean on here
A quick refresher on what this lesson assumes. Tap any card for a plain-language explanation and a tiny worked example.
Concepts you'll lean on
Tap any to see its definition and a worked example.
★Big idea
Attention = softmax over scaled query-key dot products, then weighted sum of values.
Without this:
Without attention, transformers don't exist. RNNs and CNNs would still be state of the art, with all the limits that implies for long-range dependencies.
Attention is the operation that powers every modern transformer — the architecture behind GPT, Claude, BERT, and essentially every state-of-the-art language and vision model. At its mathematical core, attention is astonishingly simple: a soft-weighted lookup in a learned key-value store. Once you see it as scaled dot-products plus a softmax, the mystique evaporates and the linear-algebra structure becomes obvious.
Start with three sets of vectors derived from the input: queries , keys , and values . Each query asks 'which keys best match me?' by computing dot products — a similarity score for every key. Softmax these scores into a probability distribution over keys, then take the value-weighted average. The whole operation is one formula:
Attention — token by token
\mathrm{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V})=\mathrm{softmax}\!\left(\dfrac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d_k}}\right)\mathbf{V}
soft lookup outputrow-wise probabilitiesscaled scoresvalues to mix
Each query gets a probability over keys, then mixes the values.
Why divide by ? When have iid components with unit variance, — dot products grow linearly with dimension. Without scaling, large pushes softmax into saturated regions where one entry is and the rest , killing gradients. Dividing by keeps the dot-product variance constant () regardless of dimension.
Why divide by $\sqrt{d_k}$
1/4
- 1Each component is iid with mean 0 and variance 1.
- 2Independence + zero mean → product variance is 1.
- 3Sum of iid unit-variance terms.
- 4Dividing by restores unit variance.
How softmax saturates without $\sqrt{d_k}$ scaling
Sigmoid is the binary case of softmax. As inputs grow, the slope flattens and gradients vanish.
↻
Multi-head attention replicates this operation times in parallel with different projections, then concatenates: . Each head learns to attend to a different relational pattern — syntax, semantics, position. Total compute is the same as one full-dimension head, but the model gets multiple specialized attentions.
Two structural facts shape what attention can and cannot do. Attention is permutation-equivariant: shuffling the rows of shuffles the rows of the output identically, so vanilla attention has no notion of order. Transformers fix this with positional encodings. Attention is also quadratic: costs — fine for small contexts, bottleneck for million-token inputs. The race for sub-quadratic attention (Performer, FlashAttention, Mamba) is the most active topic in efficient ML.
Worked example — single-query attention: , two keys , two values , . Scores: . Scaled: . Softmax: . Output: .
Scaled dot-product attention from scratch
Python (in browser)
Expected: Each row of weights sums to 1; output is shape (4, 8)
main.pypython
Loading editor…
Ready — click Run to execute in your browser.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Causal mask for autoregressive models
Python (in browser)
Expected: Lower-triangular weight matrix — token i never attends to j>i
main.pypython
Loading editor…
Ready — click Run to execute in your browser.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Building scaled dot-product attention
Same formula, four progressive lenses.
Worked example — variance check on the scaling: Generate vectors with iid entries. Without scaling, has variance → std . Differences are common, saturating softmax. With , std drops to . This single scalar made attention trainable.
In the wild
Where this shows up in production
Each card is a real system, model, or technique that rests on this lesson.
Vision Transformer (ViT)
ViTCLIPVision
Slice an image into 16×16 patches, treat each as a token, apply attention. Matches CNNs on ImageNet at scale and is the backbone of SAM, DINO, CLIP.
Cross-attention in encoder-decoder models
T5BARTSeq2Seq
T5, original Transformer, BART use cross-attention where queries come from the decoder, keys/values from the encoder — the mechanism behind translation and summarization.
Diffusion models with attention
DiffusionStableDiffusion
Stable Diffusion, Imagen, and DALL·E all use attention blocks inside the U-Net denoiser. Cross-attention from text embeddings is what makes them prompt-conditioned.
Exercises
Concept Challenge
Put your understanding to the test. Score + streak + speed all count.
Attention: Q, K, V and Scaled Dot-Product
5 rounds1000 max pts
Active recall
Recap flashcards
1 / 3
Concept
Attention = softmax over scaled query-key dot…
Tap the card to reveal
Confirm you've got it
3 quick questions. Get 2 right to mark this lesson complete.