18 · The Deep RL landscape
A field map: value-based (DQN, Rainbow), policy-gradient/actor-critic (A2C, PPO, SAC), model-based (MuZero), and RLHF — the bridge from RL to modern LLMs.
Modern deep RL splits into three families — value-based (DQN/Rainbow), policy-gradient & actor-critic (A2C, PPO, SAC), and model-based (Dyna, MuZero) — and PPO's clipped objective, the workhorse of the field, is also the algorithm behind RLHF that aligns large language models.
Without this:
Without the map you can't tell which algorithm fits a problem (discrete vs continuous actions, sample budget, simulator available) or see why RL suddenly matters for LLMs.
You now have the whole toolkit — bandits, MDPs, dynamic programming, Monte Carlo, TD, Q-learning, policy gradients, actor-critic, and DQN. This lesson is the map that places every modern method in its family.
1 — Value-based. Learn Q(s,a), act greedily. DQN is the prototype; the Rainbow paper combined six independent improvements — Double DQN (reduce overestimation), dueling networks (separate state-value and advantage), prioritised replay (sample surprising transitions more), multi-step returns, distributional RL (C51, learn the distribution of returns), and noisy nets (learned exploration). Best for discrete action spaces.
2 — Policy-gradient & actor-critic. Directly parameterise and improve the policy. A2C is the synchronous actor-critic from the last chapter; PPO (Proximal Policy Optimisation) makes policy-gradient stable by clipping how far the policy can move per update — it is the default, robust choice across most domains. SAC (Soft Actor-Critic) adds an entropy bonus for exploration and excels in continuous control (robotics). These handle continuous actions naturally, where a max over actions is intractable.
3 — Model-based. Learn (or are given) a model of the environment and plan with it. Dyna mixes real and imagined experience; MuZero learns a latent dynamics model and plans with Monte-Carlo Tree Search, mastering Go, chess, shogi, and Atari without being told the rules. Model-based methods are far more sample-efficient when a model can be learned.
The single most important algorithm to know is PPO, and the demo makes its core idea concrete: the clipped surrogate objective. PPO maximises r(θ)·A where r(θ) is the probability ratio (new policy / old policy) and A is the advantage — but it clips r to [1-ε, 1+ε] so a single update can never move the policy too far. That clip is the entire reason PPO is stable.
Python (in browser)
PPO's clipped surrogate objective in NumPy: clipping the probability ratio to [1-ε, 1+ε] caps how far one update can move the policy, which is exactly what makes PPO stable.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
In PPO, what does clipping the probability ratio r(θ) to [1-ε, 1+ε] achieve?
- Deep RL has three families: value-based (DQN, Rainbow), policy-gradient/actor-critic (A2C, PPO, SAC), and model-based (Dyna, MuZero).
- PPO is the robust default: clipping the probability ratio to [1-ε, 1+ε] bounds each policy update and keeps training stable.
- RLHF aligns LLMs with RL — the LLM is the policy, tokens are actions, a reward model scores outputs, and PPO does the optimisation.
PPO trains robots, game agents, and (via RLHF) the instruction-tuned LLMs you use daily; choosing a family is a core design decision in any RL project.
If you remove it: Without the landscape you'd reinvent algorithms blindly and miss that the same RL machinery powers both robotics and LLM alignment.