Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Environments, Gymnasium and Applications

Reward Design and Shaping

Learn how to design rewards: sparse versus dense rewards, reward hacking, potential-based shaping that keeps the optimal policy, and runnable NumPy experiments that show the effect.

The reward is the only thing an RL agent is trying to maximize, so it is the single most important design decision after the environment itself. A reward is a specification: you do not tell the agent what to do, you tell it what you want, and the agent finds whatever behavior scores highest. If your specification has a loophole, the agent will find it.

This lesson covers the trade-off between sparse and dense rewards, the many ways rewards go wrong, and a safe form of help called potential-based shaping that speeds up learning without changing what the best behavior is.

Sparse versus dense rewards

A sparse reward arrives only when the task is finished: +1 for solving the maze, 0 otherwise. It is easy to specify and hard to hack, but the agent may wander for a very long time before it sees its first reward, as in the exploration lesson. A dense reward gives feedback every step (for example, the negative distance to the goal). It guides learning, but every term you add is another thing the agent can exploit.

Sparse Dense
Specification simple and faithful needs hand-design
Learning speed slow (credit assignment, exploration) faster
Risk agent never finds the reward agent finds a shortcut the designer did not intend

Reward hacking

Agents optimize exactly what you wrote. Famous stories repeat the same pattern:

  • A boat-racing agent learned to circle in a lagoon collecting respawning bonus targets instead of finishing the race, because the score counted targets.
  • A simulated robot rewarded for "forward velocity of the head" learned to grow tall and fall over.
  • A cleaning robot rewarded for "no visible mess" learned to cover the mess or close its camera.

These are all the same failure: the reward is a proxy for what you want, and the agent maximizes the proxy. This is also known as specification gaming or Goodhart's law: when a measure becomes a target, it stops being a good measure.

Design tips that help:

  1. Reward the outcome you care about, not a behavior you believe leads to it.
  2. Keep terms few and the scales comparable; a large bonus will dominate a subtle one.
  3. Add a small per-step cost to encourage finishing quickly, but not so large that dying early looks attractive.
  4. Watch the agent: always look at behavior videos or traces, not only the reward curve.
  5. Test with an adversarial mindset: how would you get the highest score while ignoring the intent?

Potential-based reward shaping

Sometimes you do want to add guidance to a sparse reward. Doing it naively can change the optimal policy. Potential-based shaping is the principled way (Ng, Harada and Russell, 1999). Choose a potential function Phi(s), a number saying how promising a state is (for example, minus the distance to the goal). Add to each reward the term:

TEXT
F(s, a, s') = gamma * Phi(s') - Phi(s)
r_shaped = r + F(s, a, s')

The theorem: with this form of shaping, the optimal policy of the shaped problem is exactly the same as the original. The reason is that along any trajectory the shaping terms telescope: sum_t gamma^t * (gamma * Phi(s_t+1) - Phi(s_t)) collapses to gamma^T * Phi(s_T) - Phi(s_0), which does not depend on the path taken. Moving "uphill" in Phi earns a bonus, but moving back "downhill" costs the same bonus again, so you cannot farm it by wandering.

Python Naive bonus versus potential-based shaping: farming a loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the naive "closer" bonus the oscillating agent earns about six times more than the one that actually finishes: a perfect reward hack. With potential-based shaping the straight path earns the full bonus (the potential rises from -5 to 0), while the oscillating agent collects only a small, bounded amount that does not grow with the number of loops, because every step forward is paid back by the step back. Dawdling can never beat finishing.

Shaping keeps the optimal policy

Now let us check the theorem numerically. The environment is a 5-by-5 grid with a goal (+1) and a pit (-1). We solve it exactly with value iteration twice: once with the original rewards and once with potential-based shaping (Phi is minus the Manhattan distance to the goal, scaled). The theorem predicts that the optimal action values differ by exactly Phi(s), so the greedy policies are identical.

Python Shaping changes the values but not the optimal policy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two policies match cell for cell, and the shaped values equal the original values minus Phi(s) up to numerical error. Note that shaping does not change what is optimal; its benefit is in learning speed. A sample-based learner such as Q-learning receives a useful signal on every step instead of waiting for a rare goal reward. How much it helps depends on the algorithm, the exploration scheme and the potential; a poorly chosen potential, or one that is badly scaled against the true reward, can even slow learning down, so always compare against the unshaped baseline on your own problem.

Other ways to help the agent

  • Curriculum learning: start with easy versions of the task (goal close to the start) and make it harder as the agent improves.
  • Intrinsic motivation: add a bonus for novelty or surprise, such as count-based bonuses or prediction error of a learned model, when the extrinsic reward is rare.
  • Demonstrations and imitation: start from expert examples (behavior cloning), then refine with RL.
  • Inverse reinforcement learning: infer the reward from expert behavior instead of writing it by hand.
  • Learning from preferences: train a reward model from human comparisons of behavior (the idea behind RLHF).
  • Reward scaling and clipping: many deep RL setups divide rewards by a running standard deviation or clip them to [-1, 1] to keep gradients well behaved.

Common mistakes

  • Adding a bonus for "progress" that can be earned repeatedly (a loop), instead of a potential difference.
  • Shaping with Phi(s') - Phi(s) and forgetting the gamma in gamma * Phi(s'); the policy-invariance proof needs it.
  • Giving terminal states a non-zero potential. Use Phi = 0 at terminal states, or the telescoping argument fails.
  • Never looking at what the agent actually does.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is reward hacking?

    The reward is a proxy; agents exploit any gap between the proxy and the real goal.
  2. 2. What form does potential-based shaping take?

    The difference of potentials telescopes along any path, so it cannot change which policy is optimal.
  3. 3. Why can't an agent farm a loop under potential-based shaping?

    Shaping terms around a loop cancel (apart from a discount effect), so there is no net profit.
  4. 4. When is a sparse reward most problematic?

    Without any rewards to learn from, value estimates stay flat and exploration is mostly random.

Summary

  • The reward is a specification; agents maximize exactly what you write, loopholes included.
  • Sparse rewards are faithful but hard to learn from; dense rewards guide learning but invite hacking.
  • Potential-based shaping gamma * Phi(s') - Phi(s) speeds up learning while leaving the optimal policy unchanged.
  • Curricula, intrinsic motivation, demonstrations and learned reward models are alternatives for hard problems.
  • Always inspect behavior, not just reward curves.

Next: Where RL Is Used: Applications, Safety and Beyond