36 · Capstone: a tiny GPT in NumPy
Build a decoder-only GPT FORWARD PASS in pure NumPy — embeddings, masked attention blocks, and greedy decoding — exactly what runs inside GPT at inference time.
A decoder-only GPT forward pass is just: embed + positional encode → stacked masked-attention blocks → final LayerNorm → linear head → logits. Argmax the LAST position to get the next token. We build all of it in NumPy.
Without this:
Without building the forward pass yourself, GPT stays a black box; with it, you've seen the exact mechanism that runs inside every large language model at inference.
This is the capstone of the chapter — and, really, of the whole arc from the perceptron in lesson bc-dl-01 to here. We are going to build a tiny decoder-only GPT forward pass in pure NumPy.
To keep it honest about what's possible inside Pyodide's budget, we do inference only: a forward pass and greedy decoding. There is no training — no backprop, no gradient descent — because that would be far too heavy here. But that's the beautiful part: what we build IS, mechanically, exactly what runs inside GPT when you type a prompt and it streams back text. A trained GPT is this same forward pass with weights that have been fit to trillions of tokens; an untrained one (random weights, like ours) runs the identical computation but produces gibberish. The mechanism is real; only the weights are random.
The model is decoder-only (like every modern LLM): there is no encoder and no cross-attention — just a stack of masked self-attention blocks. Each block is:
x → MaskedSelfAttention → Add & Norm → FFN → Add & Norm
The full forward pass is:
tokens → token embedding + positional encoding → N masked-attention blocks → final LayerNorm → Linear head → logits (seq, vocab)
To predict the next token you take the logits at the last position and pick the argmax (greedy) — or sample from the softmax for variety. Then you append that token and run the whole thing again. That loop is autoregressive generation. Let's build it.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
PyTorch read-along: the same tiny GPT as an nn.Module — nn.Embedding (token + position), a ModuleList of masked-attention blocks, final LayerNorm, Linear head, and a greedy generate method
In a decoder-only GPT, what produces the next-token prediction?
- A decoder-only GPT forward pass is: token embedding + positional encoding → N masked self-attention blocks → final LayerNorm → linear head → logits (seq, vocab).
- To generate, take the logits at the LAST position, argmax (greedy) or sample for the next token, append it, and repeat — that loop is autoregressive decoding.
- Untrained random weights run the identical computation as a real GPT but produce gibberish: the mechanism is real, only the weights differ — scale + training is what turns this into GPT/LLaMA/Claude.
This forward pass is literally what executes inside GPT, LLaMA, Mistral, and Claude every time they generate a token. Inference servers run exactly this loop (plus KV-caching and sampling) at billion-parameter scale.
If you remove it: You'd understand transformers in theory but never have seen the end-to-end generation loop run, leaving 'how does GPT actually produce text?' as an abstraction rather than code you've built and executed.