Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Function Approximation and Deep RL

DQN Improvements: Double, Dueling, Prioritized Replay

Learn the main upgrades to Deep Q-Networks: Double DQN, dueling heads, prioritized experience replay, n-step returns, noisy nets and Rainbow, with small runnable NumPy demos.

The original DQN from the previous lesson works, but researchers quickly found specific, fixable weaknesses. Each fix is a small change to one part of the algorithm: the target, the network head, the replay sampling, the return, or the exploration. Combined, they became Rainbow, a DQN that is much stronger than the original.

This lesson explains each idea, shows the key computation in runnable NumPy, and tells you which ones are worth reaching for first.

Double DQN: removing overestimation

You met maximization bias in Expected SARSA and Double Q-learning: taking a max over noisy estimates is biased upward. A neural network is noisy everywhere, so DQN inherits the problem. Double DQN decouples selection from evaluation using the two networks DQN already has:

TEXT
DQN target:         y = r + gamma * max_a' q_target(s', a')
Double DQN target:  y = r + gamma * q_target(s', argmax_a' q_online(s', a'))

The online network chooses the action, the target network scores it. It costs one extra forward pass and no new parameters. Let us measure the bias: all ten actions truly have value zero, and each network's estimate is noisy.

Python Overestimation: DQN target vs Double DQN target

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

DQN's target is about +1.5 above the truth for ten actions; Double DQN's is close to zero. In real networks the two estimates are correlated (the target is an older copy of the online net), so Double DQN reduces the bias rather than removing it completely. It is still nearly always worth using.

Dueling networks: value plus advantage

In many states, the choice of action barely matters (an empty road with no cars). A standard Q-head must still learn a separate value for each action in each state. A dueling network splits the head into two streams that share the same body:

  • a state value V(s): a single number, how good is this state;
  • an advantage A(s, a): how much better is action a than average.

They are recombined as

TEXT
Q(s, a) = V(s) + A(s, a) - mean_a' A(s, a')

Subtracting the mean makes the split unique (otherwise you could add a constant to V and subtract it from A). Every update to any action's Q-value now also improves V(s), so the state's value is learned from many more samples.

Python The dueling head and why the mean is subtracted

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the mean of Q equals V, the value stream really does learn the average action value, and the advantage stream only has to learn relative differences.

Prioritized experience replay

Uniform replay treats every stored transition as equally useful. But a transition where the network was badly wrong (large TD error) teaches more than one it already predicts well. Prioritized experience replay (PER) samples transition i with probability

TEXT
P(i) = p_i^alpha / sum_k p_k^alpha        where p_i = |TD error_i| + small constant

alpha between 0 (uniform) and 1 (fully proportional) controls how strongly priorities matter. Sampling unevenly biases the gradient, so each sampled transition is multiplied by an importance weight w_i = (N * P(i))^(-beta), normalized by the largest weight. beta is annealed from about 0.4 to 1 during training. New transitions get the maximum priority so every one is seen at least once.

Python Prioritized replay: probabilities and importance weights

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The hard transitions appear in batches far more often, but their gradient contribution is scaled down by a small importance weight, which corrects the bias the uneven sampling introduced. A production implementation stores priorities in a sum tree so sampling and updating cost O(log N).

N-step returns

One-step targets propagate reward slowly and rely heavily on a (noisy) bootstrapped estimate. N-step targets sum n real rewards before bootstrapping:

TEXT
y = r_t + gamma * r_(t+1) + ... + gamma^(n-1) * r_(t+n-1) + gamma^n * max_a q_target(s_(t+n), a)

Reward information spreads n steps per update, usually speeding learning with n between 3 and 5. The price is variance and a subtle bias when behavior is off-policy (the intermediate actions came from an older policy).

Other upgrades

  • Noisy networks replace epsilon-greedy by adding learned noise to the layer weights; the agent explores in a state-dependent way and the noise level is learned.
  • Distributional RL (C51, QR-DQN) learns the whole distribution of returns, not just their mean. Predicting a distribution gives richer learning signals and usually better scores.
  • Rainbow combines Double DQN, dueling, prioritized replay, n-step returns, noisy nets and distributional RL in one agent. In ablations the biggest contributions come from prioritized replay and n-step returns.
  • Huber loss (smooth L1) replaces the squared error so rare huge TD errors do not produce huge gradients; this is the default in most DQN code.
flowchart TD
    DQN["DQN"] --> D["Double DQN: better targets"]
    DQN --> U["Dueling: better head"]
    DQN --> P["Prioritized replay: better sampling"]
    DQN --> N["N-step: better returns"]
    DQN --> Z["Noisy nets: better exploration"]
    D --> R["Rainbow"]
    U --> R
    P --> R
    N --> R
    Z --> R

Each improvement touches one component; Rainbow stacks them.

Which one first?

  1. Start with the plain DQN and a Huber loss, and get it working.
  2. Add Double DQN: a one-line change that rarely hurts.
  3. Add n-step returns (n = 3) and prioritized replay if sample efficiency matters.
  4. Try dueling and noisy nets when the basics are stable.

Common mistakes

  • Selecting and evaluating with the same network and calling it Double DQN.
  • Forgetting importance-sampling weights when using prioritized replay.
  • Using n-step returns without cutting the sum at the end of an episode.
  • Updating priorities with the old TD errors instead of the freshly computed ones after a gradient step.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does Double DQN compute its target?

    Separating selection from evaluation reduces the upward bias of the max operator.
  2. 2. Why does a dueling network subtract the mean advantage?

    Without it, a constant could move between V and A without changing Q.
  3. 3. What problem do importance-sampling weights solve in prioritized replay?

    Frequent sampling of high-error transitions would skew the gradient; weights w = (N P)^(-beta) correct for that.
  4. 4. What does an n-step return do compared with a one-step target?

    It trades a little variance for faster propagation of reward and less dependence on the estimated value.

Summary

  • Double DQN selects with the online net and evaluates with the target net to reduce overestimation.
  • Dueling heads learn V(s) and relative advantages, sharing learning across actions.
  • Prioritized replay samples surprising transitions more often and corrects with importance weights.
  • N-step returns, noisy nets and distributional values add further gains; Rainbow combines them.
  • Value-based methods are not the only path: policy gradients optimize the policy directly.

Next: The Policy Gradient Theorem and REINFORCE