16 · Actor-critic
REINFORCE is noisy. Add a learned value baseline (the CRITIC) and have the policy (the ACTOR) update on the ADVANTAGE = return − baseline. Same direction on average, far steadier learning.
Actor-critic keeps the policy gradient but replaces the raw return with the ADVANTAGE A = G_t − V(s_t), where a learned critic V estimates the baseline. Subtracting a state-dependent baseline leaves the gradient's expected direction UNCHANGED while sharply cutting its variance — so learning is faster and steadier.
Without this:
Without a baseline, policy gradients are too noisy to scale: every reward (even a constant offset) inflates the gradient. The actor-critic split — a policy that acts and a value function that judges — is the architecture behind A2C, A3C, and PPO.
REINFORCE's weakness is variance. The update weight is the raw return G_t, and returns swing a lot between episodes — so the gradient estimate is noisy and learning is jittery. The fix is one of the most important ideas in RL: subtract a baseline.
Replace G_t in the update with the advantage:
A_t = G_t − b(s_t)
where b(s_t) is some function of the state only. The key fact (it falls out of the policy-gradient theorem) is that subtracting a state-dependent baseline does not change the expected gradient — because E[∇log π(a) · b(s)] = b(s) · ∇ Σ_a π(a) = b(s) · ∇1 = 0. So the average update direction is identical, but a good baseline dramatically reduces the variance of each sample.
The best baseline is the state-value function V(s) — the expected return from s. Intuitively, the advantage A = G − V(s) asks: "was this action better or worse than what I'd normally expect from this state?" If an action did better than average, A is positive (make it more likely); worse than average, A is negative (make it less likely). The raw return alone can't tell "good action" from "good state".
This gives the actor-critic architecture, two cooperating learners:
- The actor is the policy π_θ — it chooses actions and is updated by
θ ← θ + α · A_t · ∇log π(a_t|s_t). - The critic is the value estimate
V_w(s)— it learns to predict returns (V(s) ← V(s) + β·(G_t − V(s))) and supplies the baseline.
Below we run plain REINFORCE and actor-critic across 30 independent seeds on the same corridor and compare how consistently each reaches a good policy. The actor-critic runs cluster far more tightly — that consistency is the variance reduction in action.
Python (in browser)
REINFORCE vs actor-critic across 30 seeds: similar final return, but the value baseline makes actor-critic reach it with much less spread (lower variance).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
In real deep RL the actor and critic are neural networks sharing the same loop. The actor net outputs action logits (a softmax policy); the critic net outputs a single V(s). Each step you compute the advantage from the critic, take a gradient ascent step on the actor's log-prob weighted by that advantage, and a gradient descent step on the critic's squared TD/MC error. The read-along below sketches one PyTorch update — the same two-update structure as the NumPy cell, just with autograd doing the gradients.
Real actor-critic in PyTorch: one shared body, an actor head (policy logits) and a critic head (V(s)). The advantage G − V(s) weights the actor loss; MSE trains the critic. Same two updates as the NumPy cell, with autograd.
Why does subtracting a state-dependent baseline V(s) from the return in the policy gradient reduce variance without introducing bias?
- Actor-critic = policy gradient with a learned value baseline: the actor updates on the advantage A = G_t − V(s_t) instead of the raw return.
- A state-dependent baseline leaves the expected gradient unchanged (no bias) but sharply reduces variance — so learning is steadier and more reliable across runs.
- Two cooperating learners: the actor (policy) acts and is reweighted by the advantage; the critic (V) learns to predict returns and supplies the baseline. This is the architecture under A2C/A3C/PPO.
Advantage-based actor-critic is the backbone of A2C, A3C, and PPO — the algorithms that train game-playing agents, robotic controllers, and the policy step of RLHF/GRPO for aligning LLMs.
If you remove it: Without a baseline/advantage, policy gradients are too high-variance to train large neural policies — every constant in the reward inflates the gradient and learning stalls.