5 · Exploration vs exploitation
The central dilemma of RL: exploit what looks best now, or explore to discover something better? Meet ε-greedy and UCB — and see why pure greed gets permanently stuck.
Exploitation picks the action that looks best given current estimates; exploration tries other actions to improve those estimates. ε-greedy explores at random a small fraction ε of the time; UCB explores by an optimism bonus that favours under-sampled arms. You must do both to find — and then keep playing — the optimal action.
Without this:
Without managing the explore/exploit trade-off, an agent either never discovers the best action (pure greed) or never cashes in on it (pure exploration) — the failure mode behind most stuck RL agents.
Last lesson exposed the problem; this lesson solves it. The exploration/exploitation dilemma is the most fundamental tension in RL:
- Exploitation — pick the action with the highest current estimate
Q(a). This banks reward now using what you already believe. - Exploration — pick a different action to learn more, accepting possibly lower reward now in hope of finding something better.
A purely greedy agent (ε = 0) only ever exploits. The danger is concrete: suppose the truly best arm happens to pay below-average on its very first pull. The greedy agent records a low Q for it, never pulls it again, and locks onto an inferior arm forever — it can't recover because it never looks back. Pure exploration has the opposite flaw: it keeps the estimates sharp but throws away reward by playing bad arms at random.
Two classic strategies balance the two:
ε-greedy. With probability 1 - ε exploit (pull argmax Q); with probability ε explore (pull a uniformly random arm). A tiny ε (say 0.01–0.1) is usually enough to escape the greedy trap, because eventually every arm gets sampled enough for its estimate to converge.
UCB (Upper Confidence Bound). Instead of exploring blindly, UCB is smart about which arm to explore. It picks the arm maximising Q(a) + c·sqrt(ln t / N(a)): the second term is an optimism bonus that's large for arms pulled few times N(a). UCB therefore prefers arms that are either good (high Q) or uncertain (low N) — "optimism in the face of uncertainty" — and the bonus shrinks as an arm is sampled.
Below we run all four on the same 10-armed bandit and report two metrics: average reward per step, and % optimal action (how often the agent pulled the genuinely best arm). Watch ε = 0 get stuck.
Python (in browser)
ε-greedy and UCB on one 10-armed bandit. Pure greed (ε=0) gets stuck on an early favourite; even a little exploration — or UCB's directed optimism — wins on both reward and % optimal action.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Why does a purely greedy (ε = 0) bandit agent often get permanently stuck on a sub-optimal arm?
- The explore/exploit dilemma: exploit to bank reward from current estimates, explore to improve them — you need both.
- ε-greedy exploits argmax Q with probability 1−ε and explores a random arm with probability ε; even ε≈0.1 beats pure greed.
- UCB adds an optimism bonus c·sqrt(ln t / N(a)) so it explores arms that are uncertain (low N) rather than at random.
- Pure greed (ε=0) can lock onto an early unlucky estimate forever; some exploration is essential in practice.
Every value-based RL agent (Q-learning, DQN) uses ε-greedy or a related scheme; UCB and Thompson sampling drive recommender systems, ad auctions and AutoML search.
If you remove it: Without a deliberate exploration strategy, agents either never discover the best action or never reap it — the most common reason RL training plateaus.