38 · Self-supervised pretraining
The label is hidden inside the text itself, so you can train on raw text at massive scale. Three objectives — causal LM (GPT), masked LM (BERT), denoising (T5) — turn unlabeled text into supervision.
Self-supervision builds the training label OUT OF the text itself — predict the next token, fill in a masked token, or reconstruct a corrupted span — so you can learn from raw, unlabeled text at internet scale.
Without this:
Supervised learning needs human-labeled data, which is scarce and expensive. Self-supervision is what unlocked training on trillions of tokens and made modern LLMs possible.
Supervised learning needs labeled examples — and labels are expensive: someone must annotate every image, rate every review, tag every sentence. There is only so much labeled data in the world. Yet the best LLMs train on trillions of tokens. Where do all those labels come from?
The trick is self-supervision: the label is already inside the text. You take raw, unlabeled text — web pages, books, code — hide part of it, and ask the model to predict the hidden part from the rest. The "answer key" is just the original text. No human annotation is needed, so the only limit is how much raw text you can collect. This is what turned "scrape the internet" into a training signal and made pretraining at massive scale possible.
There are three dominant self-supervised objectives, each defining a family of models:
-
Causal language modeling (CLM) — the GPT objective. Predict the NEXT token given all the tokens before it. The model only ever sees the left context, which is exactly what you need for generation: produce one token, append it, repeat.
-
Masked language modeling (MLM) — the BERT objective. Randomly hide (~15% of) tokens with a special
[MASK]symbol, and predict the originals using BOTH left and right context. Because it sees the whole sentence at once, MLM builds rich bidirectional representations — great for understanding tasks, not for generation. -
Denoising (span corruption) — the T5 objective. Corrupt the input (e.g. delete spans of tokens and replace each with a sentinel marker), and have the model reconstruct the missing pieces. It is a flexible middle ground used by encoder-decoder models.
All three share the same magic: a free, automatically-derived label that lets you learn the structure of language from raw text alone.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Pretraining recipes: GPT (autoregressive next-token), BERT (masked LM with 80/10/10 + next-sentence prediction), T5 (span-corruption denoising).
How does masked-LM (BERT) differ from causal-LM (GPT)?
- Self-supervision derives the training label from the text itself, so models can learn from trillions of tokens of raw, unlabeled text with no human annotation.
- Causal LM (GPT) predicts the next token from left context only; masked LM (BERT) predicts hidden tokens from bidirectional context; denoising (T5) reconstructs corrupted spans.
- BERT's masking uses the 80/10/10 rule (80% [MASK], 10% random, 10% unchanged) to reduce the gap between pretraining and downstream use where [MASK] never appears.
Every foundation model — GPT, Claude, BERT, T5, LLaMA — is created by one of these self-supervised objectives run over enormous text corpora before any task-specific tuning.
If you remove it: You'd know LLMs are 'trained on lots of text' but couldn't explain what the training signal actually is, or why GPT generates while BERT classifies.