19 · Capstone: train an agent end to end
Put the whole track together: define a gridworld with an obstacle, train a tabular Q-learning agent from scratch, then evaluate the greedy policy's success rate and print it as arrows.
A complete RL project is three steps you now own end to end: (1) define an environment as a step() function, (2) train an agent by interaction (here, tabular Q-learning), and (3) evaluate the learned greedy policy on held-out episodes — measuring success, not just training reward.
Without this:
Without assembling the pieces yourself, the algorithms stay abstract; the capstone proves you can take a problem from environment definition to a working, evaluated agent.
This is the capstone. No new theory — just the satisfying act of assembling everything from the track into one working agent. The three steps of any RL project:
- Define the environment. A small gridworld with a wall: the agent starts top-left, the goal is bottom-right, and one cell is an obstacle it cannot enter. Reward is
+1at the goal, a small-0.04step cost everywhere else (so the agent learns to reach the goal quickly). The environment is, as always, astep(state, action)function. - Train. Run tabular Q-learning (Chapter 5): act ε-greedily, and after each transition update
Q(s,a) ← Q(s,a) + α[r + γ·max_a' Q(s',a') − Q(s,a)]. We decay ε over training so the agent explores early and exploits late. Everything is seeded so the run is reproducible. - Evaluate. Turn off exploration (act purely greedily, ε=0) and run several test episodes to measure the success rate — the fraction that reach the goal. Then print the learned policy as arrows, one per cell, so you can literally see the agent's plan flow around the obstacle to the goal.
Run it. The success rate should hit 100% and the arrows should form a clean path from start to goal, routing around the wall.
Python (in browser)
End-to-end RL: a gridworld with an obstacle, tabular Q-learning trained from scratch (seeded), then the greedy policy evaluated for success rate and printed as arrows routing around the wall to the goal.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
That single cell is a complete reinforcement-learning project — and it uses every idea in the track: the agent-environment loop and reward (Ch1), the gridworld MDP (Ch1), the discounted return via γ (Ch1), exploration vs exploitation through ε-greedy (Ch2), bootstrapping with the Bellman target (Ch3-4), and tabular Q-learning control (Ch5). Swap the table for a neural network and add replay + a target net and you have DQN (Ch7); parameterise the policy directly and you have policy gradients / PPO (Ch6, Ch7).
Where to go next in RL. Three concrete steps: (1) re-implement these algorithms against gymnasium (the maintained gym) on CartPole and LunarLander in a local Python environment — the same code, real environments; (2) read the Stable-Baselines3 PPO/SAC/DQN implementations to see production-grade versions of what you built; (3) for the LLM connection, follow RLHF / DPO and the deep-learning track's instruction-tuning lesson. The conceptual foundation you now have — MDPs, returns, value functions, the Bellman equation, exploration, and the policy/value duality — transfers unchanged to every one of them.
Why is the agent evaluated with a purely greedy policy (ε = 0) on separate test episodes, rather than reporting the reward earned during training?
- A full RL project is three steps: define the environment (step function), train by interaction (Q-learning), and evaluate the greedy policy.
- Decay ε over training (explore early, exploit late) and always evaluate with ε=0 on separate episodes for an honest success rate.
- The same foundation — MDPs, returns, Bellman, exploration, policy/value — scales unchanged from this gridworld to DQN, PPO, and RLHF.
This define-train-evaluate loop is every RL project in industry; gymnasium + Stable-Baselines3 are the standard next tools, and the same flow powers robotics and LLM alignment.
If you remove it: Without hands-on assembly the algorithms remain isolated recipes; the capstone is what turns the track into a transferable skill.