13 · Gridworld project
Put it all together: an end-to-end Q-learning agent that learns to navigate a maze with walls, with proper exploration decay and a real evaluation — success rate and steps to goal.
A complete tabular-RL project has four parts: define the environment (states, walls, rewards, step function), train with ε-greedy Q-learning and decaying exploration, EVALUATE the greedy policy (success rate, steps-to-goal), and inspect the learned policy. Mastering this loop is the whole of model-free control.
Without this:
Without an evaluation step you only have a trained table, not evidence it works. Reporting greedy-policy success rate and path length is what turns 'I ran Q-learning' into 'the agent solves the maze'.
This capstone for the chapter assembles everything into one self-contained agent. The task is a 6×6 maze with walls — harder than the open grids so far, because the agent must learn to route around obstacles, and bumping a wall wastes a step. The structure mirrors any real tabular-RL project:
-
Define the environment. A text
LAYOUT(.floor,#wall,Sstart,Ggoal) compiles into astep()function. Walking into a wall or off the edge leaves you in place and still costs −1; reaching the goal gives +10. -
Train. Q-learning with ε-decay: start near-fully exploratory (ε ≈ 1) and anneal toward ε ≈ 0.05. Early on the agent must explore widely to find the goal at all; later it should mostly exploit what it has learned. Decaying ε is the standard way to get both.
-
Evaluate. Training reward is noisy because of exploration. The honest measure is the greedy policy with exploration turned off: run it many times and report the success rate (does it reach the goal?) and the average steps to goal. We compare the latter against the Manhattan-distance lower bound to confirm the path is near-optimal.
-
Inspect. Print the greedy action in every cell as an arrow, drawn over the maze, so you can literally read the policy and see it flow around the walls toward
G.
This is exactly the workflow you would follow on a larger problem — only the size of the table (or, for DQN, the network) changes.
Python (in browser)
End-to-end tabular RL: define maze → train Q-learning with ε-decay → evaluate the greedy policy (100% success, near-Manhattan path) → read the policy as arrows weaving around the walls to the goal.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Why decay ε from ~1.0 down to ~0.05 over training, rather than keeping it fixed?
- A full tabular-RL project = define environment (step function) → train with ε-greedy Q-learning + ε-decay → evaluate the greedy policy → inspect it.
- Decaying ε balances exploration (find the goal early) with exploitation (refine and act near-optimally late).
- Evaluate with exploration OFF: report greedy success rate and steps-to-goal (vs the Manhattan lower bound), not the noisy training reward.
This exact pipeline — env, ε-decay training, held-out greedy evaluation, policy inspection — is how every RL result (from gridworlds to DQN on Atari) is produced and reported.
If you remove it: Without disciplined evaluation you can't tell a working agent from one that memorised noise, and without ε-decay an agent often fails to explore enough to find sparse goals.