12 · SARSA (on-policy)
Change one symbol — max becomes the action you actually take next — and Q-learning becomes SARSA. That single change makes the agent learn a safer, more cautious policy.
SARSA updates Q(s,a) ← Q(s,a) + α·[r + γ·Q(s',a') − Q(s,a)], where a' is the action the agent ACTUALLY takes next (chosen ε-greedily), not the max. It is on-policy: it evaluates the policy it is following, including its exploration — so it learns to avoid disasters that exploration would cause.
Without this:
Without the on-policy view you'd think Q-learning's optimal-but-risky path is always preferable. SARSA shows that when exploration is dangerous (real robots, finance) the policy that accounts for its own mistakes can be the better one to deploy.
Q-learning and SARSA are twins that differ in exactly one term. The name SARSA spells out the tuple it uses for each update: State, Action, Reward, next State, next Action — (s, a, r, s', a'). Compare the targets:
- Q-learning (off-policy):
r + γ·**max**_a' Q(s', a')— assumes optimal next action. - SARSA (on-policy):
r + γ·Q(s', **a'**)— uses the action the agent actually picks next.
That a' in SARSA is chosen by the same ε-greedy policy the agent is following. So SARSA evaluates and improves the policy it is actually running, exploration included. This is what on-policy means: the policy being learned is the policy being used to generate data.
Why does this matter? Because Q-learning's target is blind to the agent's own exploratory mistakes. It learns "the value of acting optimally from here", but the agent doesn't act optimally — sometimes it explores and falls off a cliff. SARSA feels those occasional disasters (an exploratory step into the cliff contributes its −100 to the next-action value) and therefore learns a policy with a margin of safety.
The canonical demonstration is cliff-walking: a grid where the shortest path to the goal runs right along the edge of a cliff. Step into the cliff and you get −100 and reset to start. Below we train both algorithms on the same cliff and print the greedy path each one learns. Q-learning takes the risky optimal edge; SARSA takes the safer detour away from the cliff. Same environment, same exploration — one symbol of difference.
Python (in browser)
Same cliff, same ε-greedy exploration: Q-learning learns the risky optimal path along the edge; SARSA, feeling its own exploratory falls, learns the safer route one row up.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
In the cliff-walking task, why does SARSA learn a path further from the cliff than Q-learning?
- SARSA is on-policy TD control: Q(s,a) ← Q(s,a) + α·[r + γ·Q(s',a') − Q(s,a)], using the action a' actually taken next.
- On-policy means it evaluates the policy it is following (exploration included); off-policy Q-learning evaluates the greedy/optimal policy via the max.
- When exploration is dangerous, SARSA's safety-aware policy can be preferable to Q-learning's optimal-but-risky one — the cliff-walking lesson.
On-policy control underlies expected-SARSA, n-step SARSA, and is the conceptual ancestor of on-policy policy-gradient methods (A2C/PPO) in the next chapter.
If you remove it: Without the on-policy/off-policy distinction you can't reason about whether an agent should account for the cost of its own exploration when deciding how to act.