Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 37 ยท MDPs and Bellman Equations

Returns, Discounting and the Goal of RL

Understand the return G_t, the discount factor gamma, episodic versus continuing tasks, the recursive return formula and how gamma shapes the behavior an agent learns.

A reward tells the agent how good one step was. But the goal of reinforcement learning is not to collect one good reward; it is to do well over a whole sequence of steps. The quantity an agent actually tries to maximize is the return: a combination of all future rewards. Defining it carefully, especially the role of the discount factor gamma, explains a lot of surprising behavior in trained agents.

The return

For an episodic task that ends at time T, the simplest return is the sum of the rewards after time t:

G_t = R_{t+1} + R_{t+2} + ... + R_T

This breaks for continuing tasks, where T is infinite and the sum may diverge. The fix is to discount future rewards:

G_t = R_{t+1} + gamma * R_{t+2} + gamma^2 * R_{t+3} + ...

With 0 <= gamma < 1 and bounded rewards, the sum is always finite. The factor gamma expresses patience: a reward k steps away is worth gamma^k of its face value. Discounting also encodes a sensible uncertainty about the far future, and many real tasks (money, survival) genuinely prefer sooner to later.

The recursive form

Group everything after the first reward and you get the identity that powers all of RL:

G_t = R_{t+1} + gamma * G_{t+1}

The return now is the next reward plus a discounted return later. This makes computing returns easy: walk the episode backwards, keeping a running total. It will reappear as the Bellman equation in two lessons.

Python Computing returns backwards

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how with gamma = 0.5 the early returns are small even though the episode ends with a reward of 10: an agent that cares little about the future barely feels it.

Episodic and continuing tasks

An episode is one run from a start state to a terminal state. Episodic examples: a game of chess, one attempt at a maze. A continuing task has no natural end: process control, a trading agent, a recommendation system running for years.

A neat trick unifies both: treat a terminal state as an absorbing state that loops to itself with reward 0. Then every task is an infinite sequence, and the same formulas apply. In code we simply stop at the terminal and use value 0 beyond it.

What gamma does to behavior

gamma is part of the problem definition, not a tuning knob you can ignore. It changes what "optimal" means.

  • gamma = 0: a myopic agent that maximizes only the next reward.
  • gamma near 1: a far-sighted agent. It will accept short-term pain for long-term gain.
  • The effective horizon is roughly 1 / (1 - gamma) steps: gamma = 0.9 looks about 10 steps ahead, gamma = 0.99 about 100.

Consider a choice between a path that pays 1 right away and a path that pays 10 after five steps. Which is better depends entirely on gamma.

Python When does patience pay off?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The agent flips from option A to option B at gamma of about 0.56. If your trained agent behaves greedily and ignores long-term goals, a too-small gamma is a prime suspect. If training is unstable or slow, a gamma too close to 1 is another: far-away rewards make estimates noisy.

Reward design and the reward hypothesis

The reward hypothesis states that all goals can be expressed as maximizing the expected cumulative reward. In practice, designing the reward is the hardest part. Reward the wrong thing and the agent exploits it.

  • Reward a cleaning robot for "dirt collected" and it may spill dirt to re-collect it.
  • Give a maze agent -1 per step and it learns to hurry; give +1 only at the goal and it may wander for a long time before it ever finds it (a sparse reward problem).
  • Prefer rewards that state what you want, not how to do it. Shaping hints can help but can also create loopholes.

Average reward (a brief note)

For continuing tasks, some algorithms optimize the average reward per step instead of a discounted sum. It avoids choosing gamma but needs different mathematics. This course sticks to discounting, which is by far the most common.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which identity relates the return at time t to the return at t+1?

    The return is the immediate reward plus the discounted return from the next step on.
  2. 2. With gamma = 0.99, roughly how many steps ahead does the agent care about?

    The effective horizon is about 1 / (1 - gamma) = 100 steps.
  3. 3. What happens to an agent when gamma is 0?

    All terms with gamma raised to a positive power vanish, leaving only R_{t+1}.
  4. 4. Why is a discount below 1 convenient for continuing tasks?

    A geometric series with ratio below 1 converges when rewards are bounded.

Summary

  • The return G_t is the discounted sum of future rewards; RL maximizes its expectation.
  • Recursive form: G_t = R_{t+1} + gamma * G_{t+1}, so returns are computed backwards over an episode.
  • gamma sets patience; the effective horizon is about 1 / (1 - gamma).
  • Terminal states can be modelled as absorbing states with zero reward.
  • Reward design matters: agents optimize what you measure, not what you meant.

Next: Policies and Value Functions