Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 73 ยท Generative and Other Models

Neural Networks in Reinforcement Learning

Learn reinforcement learning basics: the agent loop, Q-learning tables, a Q-network, a policy-gradient method in NumPy, exploration and instability.

In supervised learning, someone hands you the right answer for every example. In reinforcement learning (RL) nobody does. An agent acts in an environment, sees what happens, and receives a number called a reward that says how well it is doing. Nothing says which action should have been taken. The agent must work that out from experience, often from rewards that arrive long after the actions that earned them.

That is the setting for game-playing programs, robot control, and the preference-tuning step that turns a language model into an assistant (see How Large Language Models Work). Neural networks enter RL as function approximators: a network that estimates how good each action is, or one that directly outputs which action to take. In this lesson you will build both from scratch in a tiny grid world.

The reinforcement learning loop

Agent Chooses an action in the current state.
Environment Responds with a new state and a reward.
Return The total reward the agent wants to maximize.

Rewards can be delayed, so credit assignment is the hard part.

1 / 2

The agent-environment loop

At each time step the agent sees a state s, picks an action a, and the environment returns a reward r and the next state s'. The episode ends at a terminal state or a time limit. The agent's goal is to maximize the return, the sum of rewards, usually discounted so that sooner rewards count more:

G = r_0 + gamma * r_1 + gamma^2 * r_2 + ..., with a discount factor gamma between 0 and 1 (here 0.95).

flowchart LR
    A["Agent<br/>(policy)"] -- "action a" --> E["Environment"]
    E -- "next state s', reward r" --> A

One turn of the loop, repeated until the episode ends.

Our environment is a 4x4 grid world. The agent starts at the top-left cell and can move up, down, left or right (walls stop it). Reaching the bottom-right cell gives +1 and ends the episode. A pit at row 1, column 2 gives -1 and also ends it. Every other step costs -0.04, so wandering is penalized, and the episode times out after 30 steps. First, see how a random agent fares.

Python The environment and a random agent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The random agent does badly. It averages a return of -1.178, finds the goal only 13.4% of the time and falls in the pit 67.7% of the time. A hand-written route (three steps down, three steps right) scores 0.8: five step costs of 0.04 plus the +1 at the goal. That number, about 0.8, is the best any agent can do here, so it is the target for the learning methods below.

Q-learning with a table

The Q-function Q(s, a) is the expected return if you take action a in state s and play well afterwards. If you knew Q, the best policy would be trivial: in every state, pick the action with the highest Q.

Q-learning learns Q from experience. After every step, it nudges the old estimate toward a TD target (temporal-difference target): the reward just received plus the discounted best value of the next state.

Q(s, a) += alpha * (r + gamma * max_a' Q(s', a') - Q(s, a))

Because the target contains an estimate of the next state's value, the agent bootstraps: it improves its guess using another guess, and information about the goal spreads backward one step at a time through repeated episodes.

There is one more ingredient: exploration. If the agent always took the action that currently looks best, it would never discover better routes. The standard fix is epsilon-greedy action selection: with probability epsilon take a random action, otherwise the best-known one. We start with epsilon = 1 (pure exploration) and reduce it to 0.05.

Python Tabular Q-learning

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The average return climbs from -1.28 (random behavior) to 0.79 after 600 episodes, essentially the best possible 0.8 (the small gap comes from the 5% of random moves that remain). The printed policy shows the route that was found, going right, down, down, right, down, right along the cells that avoid the pit marked X. The table holds 16 states x 4 actions = 64 numbers.

From a table to a Q-network

A table works only when the states can be listed. A game screen has millions of possible pixel combinations, and a robot's joint angles are continuous numbers. A Q-network replaces the table with a neural network that takes the state as input and outputs one Q-value per action. Because similar states give similar inputs, the network can generalize to states it has never visited.

Training turns the TD target into a regression problem: for a transition (s, a, r, s'), the network's output for action a should move toward r + gamma * max Q(s', .), using a squared error. This is the idea behind DQN (deep Q-network). Two additions make it work much better, and both are in the code below:

  • Experience replay. Store transitions in a buffer and train on random minibatches from it. Consecutive transitions from one episode are strongly correlated, and training on them in order makes the network chase its own recent experience. Random samples are closer to the independent examples that mini-batch training assumes.
  • A target network. The TD target uses a frozen copy of the network that is refreshed only every 25 updates. If the same network produced both the prediction and its own target, every update would shift the target too, like chasing a moving goalpost.

Here the state is the (row, column) pair scaled to the range 0 to 1, fed to a network with one hidden layer of 24 tanh units. The backward pass is the same chain rule you know, with the loss applied only to the Q-value of the action that was actually taken.

Python A Q-network with replay and a target network

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The average return over the first 50 episodes is -0.8 (mostly exploration), and over the last 50 episodes 0.65. That is below the 0.8 ceiling because of the remaining random moves and because 250 episodes is short. The greedy policy it learned is different from the table's but just as good: from the start it goes down three times and then right three times, which also reaches the goal in six steps. Many routes tie here. Different training runs can settle on different routes.

Policy gradients

Q-learning learns values and derives a policy. Policy-gradient methods learn the policy itself: a network outputs a probability for each action, and training makes good actions more likely. The simplest algorithm is REINFORCE:

  1. Play a whole episode with the current policy.
  2. For each step, compute the return G from that step to the end.
  3. Increase the log-probability of each action taken, in proportion to G.

The update for one step is grad = G * (one_hot(a) - probabilities) for a softmax policy, which pushes up the chosen action when G is large and positive, and pushes it down when G is negative. Moving the weights along this gradient is gradient ascent on the expected return.

Raw returns are noisy, so a common trick is to subtract a baseline, here a running average of recent returns. Actions are then reinforced only if they did better than usual, which reduces the variance of the gradient without biasing it. The "policy network" here is the smallest possible one: a one-hot encoding of the state multiplied by a weight matrix to give four logits.

Python REINFORCE with a baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The return rises from -1.16 early on to 0.74 by episode 800. At the start cell the policy has become confident: it chooses down with probability 0.85. Compared with Q-learning, REINFORCE needs whole episodes before every update, and its gradient estimates are noisy, so it learns more slowly here. The reward for policy methods is that they handle continuous actions naturally and can represent stochastic policies. Modern methods such as actor-critic algorithms combine both ideas: a policy network chooses actions and a value network supplies a better baseline.

From DQN to PPO: the modern toolbox

The two algorithms above are the seeds of nearly every modern method. Real systems add a handful of fixes, each aimed at one of the weaknesses listed in the next section. This part names them and demonstrates the two that are easiest to see.

Double DQN and Dueling DQN

A DQN target is r + gamma * max Q(s', a'). The max is taken over the same noisy estimates that it then trusts. If every action is truly worth the same but each estimate has random error, the largest estimate is biased upward, and bootstrapping spreads that optimism through the table. Double DQN splits the job in two: one network selects the best next action, the other evaluates it. Because the two networks make different errors, the evaluator is not fooled by the selector's lucky overestimate. In practice the online network selects and the target network evaluates, so it costs nothing extra.

Dueling DQN changes the network shape instead. The output layer is split into a state-value stream V(s) and an advantage stream A(s, a), recombined as Q = V + (A - mean(A)). In many states the choice of action hardly matters, and the network can learn V(s) from every transition instead of waiting until each action has been tried.

Python Why max is biased, and how Double DQN fixes it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All ten actions are truly worth 0, yet the single-estimator max target averages +1.54, an overestimate larger than the noise itself. Selecting with one estimate and evaluating with an independent one brings the average back to 0.00. The dueling combination subtracts the mean advantage so the two streams are identifiable: the mean of Q equals V.

Other DQN upgrades you will see named together in the "Rainbow" recipe are prioritized replay (sample surprising transitions more often), n-step returns (look several rewards ahead before bootstrapping), noisy layers for exploration and distributional Q-values. They are bookkeeping on top of the same loop.

Actor-critic and A2C

REINFORCE waits for the end of an episode and subtracts a crude baseline. An actor-critic method keeps two functions: the actor is the policy, and the critic is a value network V(s) trained by regression. The critic gives a much better baseline, and it lets the update use the advantage, A = r + gamma * V(s') - V(s): how much better the step turned out than the critic expected. A2C (advantage actor-critic) runs this synchronously on a batch of experience. Generalized advantage estimation (GAE) blends advantages over several steps with a parameter lambda to trade bias against variance.

PPO: the clipped update

Policy gradients have a trap: if one update moves the policy too far, performance can collapse and the data you collected is no longer representative. Proximal policy optimization (PPO) keeps each update small. After collecting a batch, it compares the new policy with the one that collected the data through the probability ratio rho = pi_new(a|s) / pi_old(a|s) and maximizes

min(rho * A, clip(rho, 1 - eps, 1 + eps) * A)

with eps around 0.2. When an action was good (A > 0) the objective stops rewarding increases in rho beyond 1 + eps; when it was bad, it stops rewarding decreases below 1 - eps. Clipped samples simply contribute no gradient. That lets PPO safely reuse the same batch for several epochs, which makes it far more sample-efficient than A2C. PPO is the workhorse of robotics and game agents, and a variant of it is the standard reinforcement step in language-model tuning.

The next example trains both on our grid world with a tabular softmax actor and a tabular critic, using GAE advantages (lambda = 0.9). A2C takes one gradient step per batch of 8 episodes; PPO takes six clipped steps per batch.

Python A2C versus PPO on the grid world

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After 60 batches the average return of the last ten iterations is 0.75, 0.74 and 0.76 for A2C and 0.79, 0.80 and 0.79 for PPO across three seeds, against a best possible 0.80. The PPO agent also ends up nearly deterministic at the start cell (about 99% down). PPO's edge comes from squeezing six updates out of each batch while the clip stops them from running away; it is a small environment, so treat the numbers as an illustration rather than a benchmark.

Continuous actions: DDPG, TD3 and SAC

A softmax over a handful of actions does not work for a robot arm with continuous torques. Two families extend the ideas above:

  • DDPG and its fix TD3 learn a deterministic actor a = mu(s) and a critic Q(s, a) that takes the action as an input. The actor is trained by following the critic's gradient with respect to the action. TD3 adds twin critics (take the smaller, which counters the overestimation seen above), delayed actor updates and noise on the target action.
  • SAC (soft actor-critic) learns a stochastic policy and adds an entropy bonus to the reward, so the agent is paid for staying random where randomness costs nothing. It is usually the most stable choice for continuous control.
Method Learns Actions Data use Typical use
DQN / Double / Dueling Q-values discrete off-policy, replay buffer Atari-style games
REINFORCE / A2C policy (+ critic) discrete or continuous on-policy, one step per batch learning the basics
PPO policy + critic discrete or continuous on-policy, several clipped steps per batch robotics, games, language-model tuning
DDPG / TD3 / SAC actor + Q critic continuous off-policy, replay buffer robot control

In a framework you rarely write these loops yourself. Libraries such as Stable-Baselines3 ship tested implementations; the code below is read-only because the packages are not available in the browser runner.

Python
# Read-only: needs `pip install stable-baselines3 gymnasium`
from stable_baselines3 import PPO

model = PPO("MlpPolicy", "CartPole-v1", clip_range=0.2, n_epochs=6, verbose=0)
model.learn(total_timesteps=50_000)

The NumPy example above is real and was run; the library snippet is only for orientation.

Why RL training is fragile

Reinforcement learning is harder to train than the supervised models earlier in this course, and the reasons are structural.

  • Moving targets. In supervised learning the labels are fixed. In Q-learning the target depends on the network itself, and in policy gradients the data depends on the current policy. Everything shifts as you learn.
  • Correlated data. Consecutive steps are similar, which breaks the independence assumed by gradient descent. Replay buffers help for off-policy methods like DQN.
  • Overestimation. Taking a max over noisy estimates is biased upward. Variants such as Double DQN reduce this.
  • Sparse and delayed rewards. If a reward arrives only at the very end of a long task, the signal for early actions is faint. Reward shaping helps but can create loopholes the agent exploits instead of solving the real task.
  • Exploration. Epsilon-greedy works in a 16-cell world; large worlds need smarter strategies.
  • High variance between runs. The same code with different random seeds can give very different results.
  • Sample inefficiency. An agent needs thousands to millions of episodes, which is cheap in a simulator and impossible on a real robot.

The last two are easy to see. The next example repeats REINFORCE with three different random seeds.

Python Same code, three seeds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three runs end at 0.26, 0.32 and -0.06 after 250 episodes. The same algorithm, hyperparameters and environment, but one seed is far behind: it is still stumbling into the pit now and then. This is why RL papers report averages over many seeds, and why a single good run proves little. When you debug an RL agent, change one thing at a time and repeat the run with several seeds before trusting any conclusion (the checklist in Debugging Training Runs still applies, plus a look at the reward curve and at what the agent actually does).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does Q(s, a) represent?

    Q is the expected total (discounted) reward starting from state s with action a and following a good policy afterwards. The best action in a state is the one with the highest Q.
  2. 2. Why does Q-learning use epsilon-greedy action selection?

    Always choosing the current best action can lock the agent into a mediocre habit. A small chance of a random action keeps exploring.
  3. 3. What are the two stabilizing tricks used when training a Q-network in the DQN style?

    Replay breaks the correlation between consecutive transitions, and the frozen target network stops the target from moving with every update.
  4. 4. In REINFORCE, what does subtracting a baseline from the return do?

    With a baseline, an action is reinforced only if it did better than usual. The expected gradient stays the same, but its noise falls.
  5. 5. Why can two runs of the same RL code give very different results?

    The data an RL agent learns from depends on its own random choices, so small differences compound. Report averages over several seeds.

Summary

  • An RL agent acts in an environment, receives rewards, and aims to maximize the discounted return; nobody tells it the correct action.
  • Q-learning updates a table of action values toward the TD target r + gamma * max Q(s', .), with epsilon-greedy exploration.
  • A Q-network replaces the table with a neural network so the agent can generalize; replay buffers and target networks keep the training stable.
  • Policy-gradient methods such as REINFORCE learn action probabilities directly, increasing the log-probability of actions in proportion to the return (minus a baseline).
  • Double and Dueling DQN reduce overestimation; actor-critic methods add a learned value baseline; PPO clips the update so a batch can be reused safely; DDPG, TD3 and SAC handle continuous actions.
  • RL is fragile: targets move, data is correlated, rewards are sparse, and results vary a lot between seeds.

Next: Classic Unsupervised Networks.