6 · The Bellman equations
Value functions measure how good a state (or a state-action pair) is. The Bellman expectation equation writes each value in terms of its successors — and turning it into an update gives us iterative policy evaluation.
The state-value V^π(s) is the expected return from s under policy π; the action-value Q^π(s,a) is the expected return from taking a in s then following π. The Bellman expectation equation links each value to its successors — V^π(s) = Σ_a π(a|s) Σ_s' P(s'|s,a)[R + γ V^π(s')] — and sweeping that equation as an update is iterative policy evaluation.
Without this:
Without value functions and the Bellman recursion you'd have to roll out and average full trajectories from every state — impossibly expensive. The Bellman equation is the recursive trick that makes evaluating and improving policies tractable.
We can now write down how good situations are. Two value functions do this, both defined as expected return (the discounted sum of future rewards from Chapter 1) under a fixed policy π:
- The state-value
V^π(s)— the expected return if the agent starts in statesand then acts according to π forever after. - The action-value
Q^π(s, a)— the expected return if the agent takes actionain statesfirst, and only then follows π.
The key relationship is the Bellman expectation equation. The return from s is the immediate reward plus the (discounted) return from wherever you land next. Taking expectations over the policy and the transitions:
V^π(s) = Σ_a π(a|s) · Σ_s' P(s'|s,a) · [ R(s,a,s') + γ · V^π(s') ]
Read it as: the value of a state is the average — over actions you might take and states you might land in — of (reward now + discounted value of the next state). This is a system of linear equations, one per state. Because the value of s is written in terms of the values of its successors, it is a recursion.
Here our environment's model (P and R) is known, so we can solve it without ever interacting — this is dynamic programming. The simplest method, iterative policy evaluation, just treats the Bellman equation as an assignment and sweeps it over all states until the values stop changing. Below we evaluate the uniform-random policy on a 4×4 gridworld.
Python (in browser)
Iterative policy evaluation: sweep the Bellman expectation backup until V converges. The value grid for the random policy already 'glows' toward the goal and 'dims' near the trap.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
In the Bellman expectation equation V^π(s) = Σ_a π(a|s) Σ_s' P(s'|s,a)[R + γ V^π(s')], why is V^π(s') (the value of the NEXT state) on the right-hand side?
- V^π(s) = expected return from s under π; Q^π(s,a) = expected return from taking a in s, then following π.
- The Bellman expectation equation writes each value as (reward now + γ × value of the next state), averaged over the policy and transitions.
- Iterative policy evaluation sweeps that equation as an update until V converges — possible only because the model (P, R) is known.
Every value-based method (policy iteration, Q-learning, DQN, actor-critic) is a Bellman backup in disguise; the expectation equation is also the target for the critic in modern deep RL.
If you remove it: Without the Bellman recursion, evaluating a policy would mean averaging full Monte Carlo rollouts from every state — far slower and impossible for continuing tasks.