33 · The encoder block
Assemble the transformer encoder block: multi-head attention, residual + LayerNorm, a position-wise feed-forward network, then another residual + LayerNorm. Stack N of them.
One encoder block = Multi-Head Attention → Add & Norm → Feed-Forward → Add & Norm. Residuals enable deep stacks, LayerNorm stabilizes, the FFN adds per-token nonlinearity.
Without this:
Without the block structure (residuals + norm + FFN) you can't stack attention into the deep transformers that actually work.
We now have the two ingredients — multi-head attention (lesson 31) and positional encoding (lesson 32). This lesson assembles them into the repeatable unit of a transformer encoder: the encoder block. One block is:
x → Multi-Head Attention → Add & Norm → Feed-Forward → Add & Norm
Three design choices make this work, and each earns its place:
-
Residual (skip) connections. Each sublayer's output is ADDED to its input:
x + Sublayer(x), not justSublayer(x). This lets the original signal — and, during backprop, the gradient — flow straight through, so we can stack dozens of blocks without vanishing gradients. (This is the same idea you saw in ResNets.) -
Layer normalization. After each residual add, LayerNorm normalizes each token vector to zero mean and unit variance (across its features), then rescales with learned
gammaandbeta. It keeps activations in a stable range layer after layer, which is what makes very deep stacks trainable. -
Position-wise feed-forward network (FFN). Attention MIXES information across positions but is, per position, a linear-ish weighted average. The FFN —
Linear → ReLU (or GELU) → Linear— is applied INDEPENDENTLY to each token (same weights for every position) and adds per-token nonlinearity and capacity. The inner dimension is typically4 × d_model(e.g. 2048 for d_model = 512): the network expands, applies a nonlinearity, then projects back.
A transformer encoder is just N identical blocks stacked (N = 6 in the original paper, 12/24/96 in modern models). Each block has its own weights; the output shape (seq, d_model) equals the input shape, so blocks compose cleanly.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Dashed green arrows are the residual connections skipping each sublayer.
PyTorch read-along: nn.TransformerEncoderLayer (one block) and nn.TransformerEncoder (stack of N)
What is the role of the residual (skip) connection in a Transformer block?
- An encoder block is Multi-Head Attention → Add & Norm → Feed-Forward → Add & Norm, and a transformer encoder stacks N identical blocks with shape (seq, d_model) preserved throughout.
- Residual connections give signal and gradient a clean bypass path (enabling deep stacks), and LayerNorm rescales each token to stable mean/variance after each residual add.
- The position-wise FFN (Linear → ReLU/GELU → Linear, inner dim usually 4·d_model) adds per-token nonlinearity, complementing attention's cross-position mixing.
The encoder block is the repeating unit of BERT, RoBERTa, ViT, and the encoder half of T5 and Whisper. 'BERT-large has 24 layers' means 24 of these blocks stacked.
If you remove it: You'd know attention but not how it is wired into a trainable deep network — leaving 'Add & Norm', 'FFN', and 'pre-norm vs post-norm' as mysterious labels in every architecture diagram.