17 · Deep Q-Networks (DQN)
When the state space is huge, a Q-table won't fit. DQN replaces it with a function approximator and adds two tricks — experience replay and a target network — that make training stable.
DQN scales Q-learning to enormous state spaces by approximating Q(s,a) with a parametric function (a neural net) instead of a table, and stabilises the otherwise-divergent training with two tricks: an experience replay buffer (break correlation, reuse data) and a slowly-updated target network (fix the bootstrap target).
Without this:
Without function approximation, RL is stuck on toy problems where every state fits in a table; without replay and a target network, naive Q-learning with a neural net oscillates and diverges.
Tabular Q-learning stores one number per (state, action) pair. That is fine for a 25-cell gridworld, but the screen of an Atari game has ~256^(210·160·3) possible states — no table on Earth holds that. The fix is function approximation: instead of a lookup table, represent the action-value as a parametric function
Q(s, a) ≈ Q_w(s, a)
where w are learnable parameters (the weights of a neural network). We never enumerate all states; we generalise across them. Deep Q-Networks (DQN) — the method that first cracked Atari from raw pixels — is exactly Q-learning with a neural net as Q_w.
The catch: plug a neural net naively into the Q-learning update and training diverges. Two ideas fix it:
- Experience replay. Store every transition
(s, a, r, s')in a replay buffer and train on random minibatches sampled from it. This breaks the strong correlation between consecutive samples (which violates the i.i.d. assumption of SGD) and lets each transition be reused many times — far more data-efficient. - A target network. The Q-learning target
r + γ·max_a' Q(s', a')itself depends on the weights we are updating — a moving target chasing itself. DQN keeps a second, frozen copy of the weights (w⁻) used only to compute the target, and copies the live weights into it every few hundred steps. A stable target = stable training.
Below we build the whole DQN machinery in pure NumPy with a linear approximator — Q_w(s,a) = w · features(s,a) — on a tiny corridor. Linear is the simplest function approximator, but the replay buffer, minibatch SGD, and target-network sync are identical to the deep version. Watch the Q-values converge.
Python (in browser)
Full DQN machinery in pure NumPy with a linear Q-approximator: replay buffer + minibatch SGD + periodically-synced target weights. Q-values converge so every state prefers moving toward the goal.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The cell above is structurally a complete DQN — only the approximator is linear. To turn it into the real thing you swap w for a neural network and the dot-product for a forward pass; replay, the max target, ε-greedy, and the periodic target sync stay exactly as written. The read-along below is the canonical PyTorch version: a small nn.Module Q-network, an MSE TD loss, an Adam optimiser, and a hard target-network sync. Read it to see how literally the NumPy machinery maps onto a deep network.
Real DQN in PyTorch: an nn.Module Q-network, replay buffer, MSE TD loss, and a hard target-network sync. Compare line-for-line with the NumPy machinery above.
Why does DQN use a separate, periodically-updated target network to compute r + γ·max_a' Q(s',a')?
- DQN = Q-learning with a function approximator (a neural net) for Q(s,a), so it scales to huge/continuous state spaces a table can't hold.
- Experience replay stores transitions and trains on random minibatches — breaking correlation between consecutive samples and reusing data.
- A frozen, periodically-synced target network supplies a stable bootstrap target so the regression doesn't chase itself.
DQN cracked Atari from pixels and seeded the whole 'deep RL' wave; replay + target networks are reused across countless value-based agents.
If you remove it: Without function approximation RL can't touch high-dimensional perception; without replay and a target net, neural Q-learning is unstable.