39 · Encoder, decoder, encoder-decoder
The same transformer block yields three families: encoder-only (BERT) for understanding, decoder-only (GPT) for generation, and encoder-decoder (T5) for seq2seq — the difference is the attention mask.
From ONE transformer block come three families: encoder-only (bidirectional → understanding), decoder-only (causal → generation), and encoder-decoder (both → seq2seq). The attention MASK is what distinguishes them.
Without this:
Knowing which family fits a task — BERT for embeddings, GPT for chat, T5 for translation — is the difference between picking the right model and fighting the wrong one.
The transformer block you built in Chapter 8 is a single, reusable unit. Remarkably, the entire modern LLM landscape comes from arranging that same block in one of three ways, distinguished mainly by which attention mask they use:
-
Encoder-only (BERT, RoBERTa, DistilBERT). Stacks of blocks with bidirectional attention — every token attends to every other token, left and right. This builds a deep, context-rich representation of the whole input at once. Perfect for understanding tasks: classification, named-entity recognition, and producing sentence embeddings for search. It is paired with the masked-LM objective (lesson 38). It does NOT generate text token-by-token.
-
Decoder-only (GPT, LLaMA, Mistral, Claude). Stacks of blocks with causal attention — each token attends only to itself and the tokens before it (a triangular mask blocks the future). This is exactly what generation needs: predict the next token, append, repeat. Paired with the causal-LM objective.
-
Encoder-decoder (the original Transformer, T5, BART). An encoder (bidirectional) reads the full input into a representation; a decoder (causal) generates the output while also cross-attending to the encoder's output. This is the natural fit for sequence-to-sequence tasks where input and output are different sequences: translation, summarization.
So the families are not three different inventions — they are three configurations of the same attention block. The masks we'll compute below (bidirectional vs causal) are literally the switch that picks a family. This connects straight back to the masked attention of lessons 30 and 34.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Dashed green arrows are the residual connections skipping each sublayer.
Dashed green arrows are the residual connections skipping each sublayer.
Task-to-family table: understanding/embeddings → encoder (BERT); chat/completion/code → decoder (GPT); translation/summarization → encoder-decoder (T5).
Which family best fits producing a single sentence embedding for semantic search?
- Encoder-only (BERT) uses bidirectional attention for understanding/embeddings; decoder-only (GPT) uses causal attention for generation; encoder-decoder (T5) combines both for seq2seq.
- All three are configurations of the same transformer block — the attention MASK (bidirectional vs causal triangular) is the core switch between encoder and decoder behavior.
- Decoder-only won the scaling race because next-token prediction at scale generalizes to almost any task via prompting, with no task-specific heads.
Choosing a model starts here: a retrieval pipeline reaches for a BERT-style encoder, a chatbot for a GPT-style decoder, a translation service for a T5-style encoder-decoder.
If you remove it: You'd treat 'transformer' as one monolithic thing and reach for the wrong model — e.g. trying to get embeddings out of GPT or generation out of BERT.