43 · Capstone: from LLM to assistant
A raw LLM only COMPLETES text; an assistant FOLLOWS instructions. Two stages bridge the gap: SFT (instruction tuning on demonstrations) then RLHF (a reward model from human preferences, optimized with PPO) — or the simpler DPO.
A pretrained LLM only COMPLETES text; turning it into a helpful, harmless assistant takes two stages: SFT (supervised instruction tuning on demonstrations) then RLHF (train a reward model from human preference comparisons, then optimize the policy against it with PPO) — DPO is a simpler alternative.
Without this:
Pretraining alone gives a text-completion engine, not ChatGPT or Claude. SFT + RLHF is the alignment recipe that makes a raw LLM follow instructions and behave helpfully and safely — the final step from this bootcamp's perceptron to a real assistant.
A raw pretrained LLM (lesson 38) has one skill: complete text. Give it "The capital of France is" and it continues "Paris." But ask it a question and it might continue with MORE questions — because internet text is full of question lists. It is not an assistant; it is an autocomplete engine. An assistant does something different: it FOLLOWS instructions, and aims to be helpful and harmless. Two stages turn one into the other — this is alignment.
Stage 1 — SFT (supervised fine-tuning / instruction tuning). Collect a dataset of (instruction, ideal response) demonstrations written by humans, and fine-tune the pretrained model on them (ordinary supervised learning, lesson 40). Now the model has learned the FORMAT of being helpful: given an instruction, produce a direct, useful answer instead of rambling. But "good demonstrations" are expensive, and they only show ONE acceptable answer per prompt.
Stage 2 — RLHF (reinforcement learning from human feedback). It is far easier for humans to COMPARE two responses ("which is better?") than to write a perfect one. So: collect many preference comparisons, and train a reward model that learns to assign a scalar score matching human preferences. Then use reinforcement learning (PPO) to optimize the SFT model — the policy — to maximize that reward, while a KL penalty keeps it from drifting too far from the sensible SFT model.
DPO (Direct Preference Optimization) is a simpler, increasingly popular alternative: it skips the separate reward model and the RL loop entirely, optimizing the policy directly on the preference pairs with a single classification-style loss. The cell below builds the heart of the reward signal — the Bradley-Terry model that converts two scalar rewards into "probability a human prefers A over B."
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
RLHF pipeline: (1) SFT on demonstrations → (2) reward model trained with the Bradley-Terry ranking loss on preference pairs → (3) PPO maximizing reward minus a KL penalty to the SFT reference. DPO skips the reward model and optimizes the policy directly on preferences.
What is the role of the reward model in RLHF?
- A pretrained LLM only completes text; alignment turns it into an instruction-following, helpful-and-harmless assistant via SFT then RLHF.
- SFT fine-tunes on (instruction, ideal response) demonstrations; RLHF trains a reward model from human preference comparisons (Bradley-Terry: P(A>B)=sigmoid(r_A−r_B)) and optimizes the policy against it with PPO + a KL penalty.
- DPO is a simpler alternative that skips the separate reward model and RL loop, optimizing the policy directly on preference pairs — and scaling laws (Chinchilla) + emergent abilities explain why this whole recipe needs scale to work.
Every chat assistant you use — ChatGPT, Claude, Gemini — is a pretrained LLM aligned with SFT + RLHF (or DPO). It is the difference between a raw autocomplete and a model that helpfully follows your instructions.
If you remove it: You'd know how to pretrain a text-completion model but not how it becomes a usable assistant — missing the alignment step that defines every product people actually chat with.