Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 37 ยท Projects and Practice

Practice: Debugging RL and Self-Test

Learn how to test and debug RL code: sanity tests on tiny MDPs, seed variance and confidence intervals, a debugging checklist, a mistakes catalog and practice exercises with answers.

Reinforcement learning has a reputation for being hard to debug, and the reputation is earned. In supervised learning a bug usually shows up as a bad loss curve. In RL a bug often shows up as an agent that is "kind of learning" (the reward goes up a little, then flat), indistinguishable from a hard problem or bad luck. The algorithm is stochastic, the data depend on the policy, and small implementation details (a missing done flag, a reward scale) decide whether it works.

The good news is that there is a discipline that finds most bugs quickly. This lesson gives you that discipline: test your code on problems with known answers, measure variance honestly, follow a checklist when things go wrong, and finish with practice questions to test what you learned in this course.

Rule 1: test on tiny problems with known answers

Never debug a new algorithm on a hard environment. First run it on problems whose optimal value you can compute by hand. If it fails there, the bug is in your code, not in the problem.

Good sanity problems:

  1. One-step bandit: two actions with rewards 0 and 1. Q should converge to [0, 1].
  2. Two-step chain with a discount: action go moves from state 0 to state 1 (reward 0) and then to the terminal state (reward 1). The optimal Q(0, go) is exactly gamma. This checks that bootstrapping and discounting work.
  3. Terminal handling: a state that ends the episode must not bootstrap beyond it.
  4. Reproducibility: the same seed must give the same numbers.
  5. Beat random: on any task, a trained agent should outperform a random policy.
Python A unit-test harness for a Q-learning implementation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last check is a negative test: it proves that the harness can detect a real bug (the missing terminal check inflates Q(1, go) above 1, because the value keeps feeding itself). A test that cannot fail is worthless.

Rule 2: one seed proves nothing

RL results vary wildly between random seeds. A method that wins on seed 0 may lose on seed 1. Papers have drawn wrong conclusions from three runs. Always run many seeds (at least 5, ideally 10 or more), report the mean with a confidence interval and compare methods with the same set of seeds.

A quick way to get an interval without assuming any distribution is the bootstrap: resample the per-seed scores with replacement many times and look at the spread of the resampled means.

Python Seed variance and bootstrap confidence intervals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the standard deviation across seeds and the width of the intervals, then at the seed-by-seed win count. If the intervals overlap heavily, or one setting wins in only some of the seeds, you do not have evidence that one is better, however different the means look.

Rule 3: follow the checklist

When an agent does not learn, go down this list in order. Stop at the first thing that is wrong.

  1. Environment: step through the environment by hand. Do rewards and done flags look right? Can a trivial policy reach the goal? Is the observation what you think it is?
  2. Reward scale: are rewards huge or tiny? Normalize or clip them. Is the reward actually obtainable?
  3. Terminal handling: is bootstrapping cut at terminated but not at time-limit truncated?
  4. Exploration: is epsilon or the policy entropy decayed too fast or too slowly? Is the agent seeing the rewarding states at all? Log state visit counts.
  5. Evaluation: are you evaluating the greedy policy, on fresh seeds, separately from training?
  6. Learning rate and target network (deep RL): too high a learning rate makes Q-values explode; log the mean and maximum of Q.
  7. Data: is the replay buffer being filled with correct tuples? Print a batch. Are shapes and dtypes right, and is the batch dimension not silently broadcasting?
  8. Gradients: is the loss going down on a fixed tiny batch (overfit one batch on purpose)? Are gradient norms finite?
  9. Compare with a known baseline: a published implementation, or a simple method like CEM from the CartPole project.
  10. Only then tune hyperparameters.

Metrics worth logging in deep RL: episode return and length, mean and max Q, TD loss, policy entropy, KL to the previous policy (PPO), fraction of clipped ratios, gradient norm and the exploration rate.

Catalog of common mistakes

Symptom Likely cause
Q-values grow without bound Bootstrapping across terminal states, learning rate too high, no target network
Reward rises, then collapses Too-large policy updates (use PPO clipping, lower learning rate)
Agent never improves over random Reward never reached (exploration), wrong sign of the update, observation not informative
Training good, evaluation bad Exploration noise during training, overfitting to a seed, evaluation using the wrong mode
Great on one seed, bad on others Variance; run more seeds before concluding anything
Agent exploits something weird Reward hacking (see reward design)
Learning stalls at a plateau Entropy collapsed, learning rate too low, replay buffer too small or stale

Practice exercises

Try these before looking at the quiz. They use only what you learned in this course.

  1. Modify the Q-learning agent from the Q-learning lesson to use Expected SARSA and compare their learning curves on the cliff-walking task over 10 seeds with confidence intervals.
  2. In the Dyna-Q maze, change the maze so that a shortcut opens after 50 episodes. How many planning steps does it take for the agent to notice? Implement the Dyna-Q+ bonus.
  3. Implement n-step TD prediction for the random walk of the function approximation lesson and plot the error against n.
  4. Add a target network and then a Double DQN target to the DQN agent. Does the maximum Q-value drop?
  5. Train REINFORCE on CartPole with a softmax linear policy using your own physics from the project, then add a baseline and measure the variance across seeds.
  6. Implement PPO's clipped objective for a Gaussian policy on a one-dimensional "reach the target" task.
  7. Build a Gymnasium environment for a game of your choice, validate it with check_env, and train a tabular agent on it.
  8. Take any environment you wrote and design three different reward functions. Describe the loophole each one invites.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should you test a new RL algorithm on tiny problems with known solutions first?

    Known optimal values give you an exact target to check against.
  2. 2. You compare two hyperparameter settings using one seed each. What is wrong?

    Use many seeds, report intervals and compare methods on the same seeds.
  3. 3. Q-values in your DQN grow without bound. Which is the most likely cause?

    These all let the target depend on a growing estimate and feed errors back into themselves.
  4. 4. What is a negative test, as used in the harness above?

    It proves the tests are able to detect the bug they are meant to detect.

Summary

  • Debug RL like a scientist: tiny problems with known answers, negative tests and fixed seeds first.
  • Run many seeds, report intervals (bootstrap is simple) and compare methods on the same seeds.
  • Work down a checklist: environment, rewards, terminal handling, exploration, evaluation, then the algorithm and hyperparameters.
  • Log the diagnostic quantities (Q statistics, entropy, KL, gradient norms), not only the return.
  • Keep practicing with the exercises; each one reuses a piece of this course.

Back to the track overview: Reinforcement Learning