32 · Positional encoding
Self-attention is order-blind: it treats a sentence as a set. Positional encoding injects sequence order back into the embeddings.
Self-attention is permutation-equivariant — it sees a sequence as an unordered SET. Positional encoding adds a position-dependent signal so the model knows token ORDER.
Without this:
Without positional encoding, a transformer cannot distinguish 'dog bites man' from 'man bites dog' — order would be invisible.
Self-attention has a property we glossed over: it is permutation-equivariant. If you shuffle the input tokens, the output tokens are shuffled the same way — but each token's computed representation does NOT change. Why? Look at the formula softmax(QKᵀ/√d_k)·V: it is built entirely from pairwise dot products and weighted sums, none of which reference WHERE a token sits in the sequence. To self-attention, a sentence is an unordered set of vectors, not a sequence.
That is catastrophic for language. "dog bites man" and "man bites dog" contain the exact same tokens; with no positional signal, self-attention produces identical representations for both. The order — the very thing that carries the meaning — is invisible. We must inject order back in.
The original Transformer (Vaswani et al. 2017) does this with sinusoidal positional encoding: a fixed (non-learned) matrix PE of shape (seq_len, d_model) that is ADDED to the token embeddings. Each position pos and each embedding dimension i gets a value from a sine or cosine of a position-dependent frequency:
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))— even dimensionsPE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))— odd dimensions
Low dimensions (small i) oscillate fast (short wavelength); high dimensions oscillate slowly (long wavelength). Together they give every position a unique, smooth, multi-frequency "fingerprint" — much like binary counting, where the low bit flips every step and high bits flip rarely. Adding this fingerprint to the embedding lets attention's dot products become position-aware.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Sinusoidal PE heatmap: low dimensions form fast stripes (short wavelength), high dimensions form slow bands (long wavelength) — each row is a unique position fingerprint
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Positional information: sinusoidal (fixed) vs learned absolute vs RoPE (rotary, relative — covered in Chapter 9)
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Why does a Transformer need positional encoding when an RNN does not?
- Self-attention is permutation-equivariant — it treats a sequence as a set — so a Transformer must inject order via positional encoding added to the embeddings.
- Sinusoidal PE uses sin on even dimensions and cos on odd dimensions across geometric frequencies, giving each position a unique multi-frequency fingerprint.
- Because it is a fixed function of position, sinusoidal PE can extrapolate to sequence lengths longer than those seen in training; learned absolute embeddings cannot.
Every transformer adds a positional signal before the first attention layer. The original Transformer and many speech models use sinusoidal PE; BERT/GPT use learned; modern LLMs (LLaMA) use RoPE.
If you remove it: You'd be confused why transformers ship a separate 'position' input at all, and unable to reason about context-length limits or why long-context tricks like RoPE scaling exist.