15 · REINFORCE
REINFORCE is the Monte Carlo policy gradient: run a whole episode, compute the return-to-go from each step, then nudge θ by ∇log π(a) × that return. Watch a softmax policy learn to walk to the goal.
REINFORCE turns the policy-gradient idea into a working algorithm: sample a full episode, compute the discounted return-to-go G_t from each step, then update θ_s ← θ_s + α · G_t · ∇θ log π(a_t|s_t) for every step. Good episodes make their actions more likely.
Without this:
Without REINFORCE you have the score function but no rule to turn real interaction into a policy update. REINFORCE is the simplest complete policy-gradient learner and the template every later method (A2C, PPO) refines.
REINFORCE (Williams, 1992) is the canonical Monte Carlo policy-gradient algorithm. "Monte Carlo" because, like the MC prediction of Chapter 4, it waits until an episode finishes and uses the actual observed return — no bootstrapping. The recipe per episode:
- Roll out a full episode by sampling actions from the current policy π_θ: record
(s_t, a_t, r_t)at each step. - Compute the return-to-go
G_tfrom each step — the discounted sum of rewards from that step onward:G_t = r_t + γ·r_{t+1} + γ²·r_{t+2} + …. Work backwards so it's one pass. - Update the preferences for every visited state:
θ_s ← θ_s + α · G_t · ∇θ log π(a_t | s_t).
Why return-to-go and not the whole-episode return? An action at step t can only influence rewards from t onward — it gets no credit (or blame) for what happened before it. Using G_t is the correct, lower-variance credit assignment.
The intuition is trial-and-reweighting: actions that were part of high-return episodes get their probability pushed up; actions from low-return episodes get pushed down. There's no model of the environment and no value table — just sampled returns reweighting the policy.
Below we run REINFORCE on a tiny 4-cell corridor (start at cell 0, goal at cell 3). The policy starts uniform; we print the rising average return over training and the final action distribution, which should converge to "go right" everywhere.
Python (in browser)
REINFORCE on a 4-cell corridor: average return climbs as the softmax policy learns, converging to P(right)≈1 everywhere (walk to the goal).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
In REINFORCE, why is each step's update weighted by the return-to-go G_t (rewards from step t onward) rather than the full-episode return?
- REINFORCE = Monte Carlo policy gradient: roll out an episode, compute return-to-go G_t per step, then θ_s ← θ_s + α · G_t · ∇log π(a_t|s_t).
- Use the return FROM each step onward (return-to-go), because an action can only affect future rewards — this is the correct credit assignment.
- It works (the corridor policy converges to 'go right') but is high variance — the motivation for baselines and actor-critic next.
REINFORCE is the simplest member of the policy-gradient family used to train policies for robotics, games, and language models; PPO is a variance-reduced, clipped descendant.
If you remove it: Without a Monte Carlo policy-gradient estimator there's no way to convert sampled episodes into a policy improvement — the score function alone can't learn.