Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 37 ยท Function Approximation and Deep RL

Deep Q-Networks from Scratch

Build a Deep Q-Network step by step in NumPy: a small neural network for Q-values, experience replay, a target network and the full DQN training loop on a runnable gridworld.

Tile coding needs you to design the features. A neural network can learn its own features from raw inputs, which is why the 2013-2015 Deep Q-Network (DQN) results, an agent learning Atari games from raw pixels, started the deep RL era. Underneath, DQN is just Q-learning where the Q-table is replaced by a neural network q(s, a; w). Two stabilizing ideas make that work: experience replay and a target network.

You will build every piece in plain NumPy so nothing is hidden. The network is small, the task is a gridworld, and the whole thing trains inside your browser.

Q-learning with a network

Tabular Q-learning moves Q(s, a) toward r + gamma * max_a' Q(s', a'). With a network we do the same by gradient descent on the squared error:

TEXT
target = r                              if s' is terminal
target = r + gamma * max_a' q(s', a'; w-)   otherwise
loss   = (target - q(s, a; w))^2

w- are the weights of the target network, explained below. The network takes the state as input and outputs one Q-value per action, so a single forward pass scores all actions.

A tiny network with backpropagation

Here is a two-layer network (input, one ReLU hidden layer, linear output) with hand-written backpropagation. Before trusting the gradient, we check it against a numerical estimate. This is a habit worth keeping: gradient checks catch the most common deep RL bug, a wrong gradient that still trains "a bit".

Python A NumPy Q-network and a gradient check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The difference should be tiny (around 1e-9 or smaller). Note how the loss gradient dq is zero for every action except the one that was taken: we only have a target for the chosen action.

Why plain online training fails

If you update the network on each new transition as it arrives, two things go wrong:

  1. Correlated data. Consecutive transitions look alike (the agent is in the same corner for ten steps). SGD assumes independent samples; correlated ones make the network chase the latest situation and forget the rest.
  2. Moving targets. The target r + gamma * max q(s', a'; w) uses the same weights being updated. Every step changes the target, which can spiral into oscillation or divergence (the deadly triad from the function approximation lesson).

DQN's two fixes:

  • Experience replay. Store every transition (s, a, r, s', done) in a big buffer. To learn, sample a random mini-batch from the buffer. Random samples are nearly independent, and each transition is reused many times.
  • Target network. Keep a second copy of the network, w-, used only to compute targets. Copy w into w- every few hundred steps. The target stays still between copies, so the regression problem is stable.
flowchart LR
    ENV["Environment"] -->|"(s, a, r, s', done)"| BUF["Replay buffer"]
    BUF -->|"random mini-batch"| TRAIN["Gradient step on online net"]
    TARGET["Target network (frozen copy)"] -->|"targets"| TRAIN
    TRAIN -->|"copy every C steps"| TARGET
    TRAIN -->|"epsilon-greedy"| ENV

Replay breaks correlation; the target network stops the target from chasing itself.

The whole agent

The environment is a 5-by-5 grid. The agent starts in the top-left corner and must reach the goal at the bottom-right (+1). A pit in the middle ends the episode with -1, and each step costs 0.04. The network sees only the normalized (row, col) position, two numbers.

Python DQN on a gridworld

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the arrow map: most arrows point toward the goal and around the pit. If the agent misses it in a few cells, change the seed or train for more episodes. With a neural network, the result varies run to run in a way a Q-table never does.

Hyperparameters that matter

Setting Typical range Effect
Learning rate 1e-4 to 1e-2 Too high: Q-values oscillate or blow up
Replay buffer size 10^4 to 10^6 Too small: forgets old situations; too big: learns from stale data
Batch size 32 to 256 Larger is smoother but slower
Target sync interval 100 to 10,000 steps Too frequent: unstable; too rare: slow
Epsilon schedule 1.0 down to 0.01-0.1 Governs exploration (see exploration strategies)
Gradient clipping clip value or norm Prevents rare huge errors from wrecking the network

Common mistakes

  • Forgetting (1 - done) in the target: the agent then "bootstraps" from the state after the episode end.
  • Computing the target with the online network and no copy, which is just unstable Q-learning.
  • Using the same transition list in order instead of sampling randomly.
  • Not normalizing inputs; features ranging over thousands next to features near 1 destroy learning.
  • Judging by the training reward with exploration still on; evaluate the greedy policy.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the purpose of experience replay?

    Randomly sampling stored transitions gives near-independent mini-batches and uses each experience many times.
  2. 2. Why is a separate target network used?

    Targets from a frozen copy stay fixed between syncs, making the regression problem more stable.
  3. 3. In the DQN loss, why is the output gradient zero for all actions except the chosen one?

    Only q(s, a) for the taken action has an observed target r + gamma max q(s', .).
  4. 4. What should the target be for a transition that ended the episode?

    There is no next state to bootstrap from, so the target is just the reward.

Summary

  • DQN is Q-learning with a neural network that maps a state to one Q-value per action.
  • Experience replay and a target network tame the instability of the deadly triad.
  • The loss touches only the taken action; terminal transitions use the reward alone.
  • Check gradients numerically, normalize inputs and evaluate the greedy policy.
  • Several refinements (Double DQN, dueling heads, prioritized replay) are next.

Next: DQN Improvements: Double, Dueling, Prioritized Replay