3 · Returns, discounting & policies
The agent maximises the RETURN — the discounted sum of future rewards. A policy is its strategy. These two definitions set up everything that follows.
The return G_t = r_{t+1} + γ r_{t+2} + γ² r_{t+3} + … is the discounted sum of future rewards the agent actually maximises. A policy π(a|s) is the agent's strategy — a mapping from states to action probabilities.
Without this:
Without a precise objective (the discounted return) and a precise strategy object (the policy), you can't compare agents or define what learning even means.
Two definitions complete the foundations.
The return. The agent doesn't maximise immediate reward — it maximises the return G_t, the (discounted) sum of all future rewards from time t:
G_t = r_{t+1} + γ·r_{t+2} + γ²·r_{t+3} + … = Σ γ^k · r_{t+k+1}
The discount factor γ (0 ≤ γ < 1) does two jobs: it makes the infinite sum finite (it converges), and it encodes how far-sighted the agent is. γ near 0 → myopic (only the next reward matters); γ near 1 → far-sighted (rewards far in the future count almost as much as now). A reward R received k steps from now is worth only γ^k · R today.
The policy. A policy π is the agent's strategy: π(a | s) is the probability of taking action a in state s. A deterministic policy always picks one action per state; a stochastic policy spreads probability over actions. Learning, in RL, means improving the policy so the expected return goes up.
Below we compute returns for the same reward sequence under different γ, and evaluate a fixed policy's expected return by simulation (a "Monte Carlo" estimate — the subject of Chapter 4).
Python (in browser)
Return = discounted sum of future rewards; a policy maps states to actions. A better policy → higher expected return, the quantity RL maximises.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
A reward of +100 will be received 10 steps from now. With discount factor γ = 0.9, roughly what is it worth to the agent today?
- The return G_t is the discounted sum of future rewards Σ γ^k r_{t+k+1} — the actual objective the agent maximises.
- γ (0≤γ<1) makes the sum finite and sets how far-sighted the agent is; a reward k steps away is worth γ^k of its value.
- A policy π(a|s) is the agent's strategy; learning = improving the policy to raise expected return.
Every RL algorithm optimises expected discounted return; γ appears in every value update and policy-gradient estimator in this track.
If you remove it: Without a defined return and policy, 'better agent' is undefined and no learning rule can be derived.