Skip to content
Tutorials Center
My learning

Topics

Lesson 90 of 111 ยท Reinforcement Learning

Policy Gradients, REINFORCE and Actor-Critic

Learn policy-based reinforcement learning: the policy gradient idea, REINFORCE with a baseline, and a one-step actor-critic, all built in NumPy on a small grid.

Q-learning learns how good each action is and then picks the best one. The policy itself is a by-product: "take the action with the highest Q". Policy-based methods skip that detour. They write the policy as a function with adjustable parameters, ฯ€(a | s; ฮธ), and change ฮธ directly so that the expected return goes up. This is policy gradient learning, and it is the foundation of most modern RL, including the methods that train large language models from human feedback.

Why bother with a second family? A policy can output probabilities, so it can be stochastic, which matters in games where predictability loses. It works naturally with continuous actions, where "take the max over all actions" is itself a hard problem. And small changes to ฮธ give small changes in behavior, while a tiny change in a Q-value can flip the greedy action. The price is noisy learning, and most of this lesson is about how to tame that noise.

Value-based versus policy-based

Value-based Learn Q, then act greedily.
Policy-based Learn the policy directly.
Actor-critic Both together.
1 / 2

The policy gradient idea

For a discrete set of actions, the usual policy is a softmax over a score (a "preference") for each action: the higher the preference, the more likely the action. In a small grid world you can keep a table ฮธ[s, a] of preferences, one row per state.

Suppose the agent played an episode and the return after taking action a in state s was G. The policy gradient theorem says that the direction that increases the expected return is, on average:

โˆ‡J(ฮธ) = E[ G ยท โˆ‡ log ฯ€(a | s; ฮธ) ]

In words: increase the log-probability of each action you took, in proportion to the return that followed. Good outcomes make their actions more likely, bad outcomes make them less likely. For a softmax table the gradient of log ฯ€(a | s) with respect to the preferences of state s has a simple form: onehot(a) โˆ’ ฯ€(ยท | s). The chosen action goes up, all the others go down a little, and the update is ฮธ[s] += ฮฑ ยท G ยท (onehot(a) โˆ’ ฯ€(ยท | s)).

This is the REINFORCE algorithm: play a whole episode, compute the discounted return G_t = r_t + ฮณ ยท G_{t+1} for every step, and apply that update at every step. It is a Monte Carlo method, since it waits for the episode to end, as in Monte Carlo versus TD learning.

REINFORCE on a small grid

The world is a 4x4 grid. The agent starts in the top-left corner, and every step costs -0.1 until it reaches the bottom-right goal, which pays +1 and ends the episode (episodes are cut off after 40 steps). The shortest path takes 6 steps. A table of 16 rows of 4 preferences starts at zero, so the first policy is uniformly random.

The code also compares learning with and without a baseline, a number b that is subtracted from the return, so the update uses G โˆ’ b. Here b is a running average of the episode returns. At the end it measures how much noise the baseline removes from the gradient of the first decision.

Python REINFORCE with and without a baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both versions learn: episodes shrink from about 16 steps at the start to about 7 later (the shortest is 6, and the policy is still stochastic, so it occasionally wanders). The learned grid shows arrows pointing down and right toward the goal. The surprise is that the baseline did not change the learning curve noticeably here. The last two lines show why it is still worth having: the average gradient estimate for the first decision is almost the same with and without it, but its variance drops from about 1.29 to about 0.28. A baseline cannot change what the gradient is on average, because it does not depend on the action. It only removes noise. On harder problems, with long episodes and large rewards, that noise is the difference between learning and not learning.

Raw REINFORCE is unbiased but noisy. Every update uses one whole episode, so one lucky or unlucky run can push the policy a long way. In larger problems the learning rate has to be small, and training is slow.

Actor-critic: a learned baseline

A running average is a crude baseline, because it is the same for every state. A better one is the value of the state itself, V(s): how much return we expect from here on. The difference G โˆ’ V(s) is called the advantage, and it says whether this action turned out better or worse than average for this state.

An actor-critic method learns both pieces at the same time:

  • The actor is the policy ฯ€(a | s; ฮธ), as before.
  • The critic is a value estimate V(s), trained with a TD update like in Q-learning.

The critic also removes the need to wait for the end of the episode. After every step, the TD error ฮด = r + ฮณ ยท V(s') โˆ’ V(s) is a one-step estimate of the advantage. The critic moves V(s) toward the target by ฮฑ_critic ยท ฮด, and the actor uses the same ฮด in place of G in its update: ฮธ[s] += ฮฑ_actor ยท ฮด ยท (onehot(a) โˆ’ ฯ€(ยท | s)). The two learn together: the critic judges the actor's choices, and the actor's choices decide what experience the critic sees.

flowchart LR
    ENV["environment"] -->|"state s"| ACTOR["actor: policy pi"]
    ACTOR -->|"action a"| ENV
    ENV -->|"reward r, next state s'"| CRITIC["critic: value V(s)"]
    CRITIC -->|"TD error delta"| ACTOR
    CRITIC -->|"TD error delta"| CRITIC

The critic turns raw rewards into a TD error that both networks learn from.

Python A one-step actor-critic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The actor-critic reaches an average episode length of about 7 steps on the final 50 episodes, with little difference between the 8 seeds. The curve of block averages shows the usual pattern: about 18 steps in the first hundred episodes, falling to roughly 7. The critic's table is easy to read as a heat map of the goal: the values rise smoothly toward the goal corner (about 1.0 beside it) and fall off to about 0.4 at the start. The updates happen at every step, so there is no need to wait for an episode to finish, which also makes the method usable in tasks that never end.

From tables to A2C and A3C

Real problems have too many states for a table, so both ฯ€ and V become neural networks (see Neural Networks in Reinforcement Learning). The same loop applies:

  • A2C (advantage actor-critic) runs several copies of the environment in lockstep, collects a short batch of steps from each, and updates the shared actor and critic from that batch. The advantage is estimated from a few steps of real rewards plus the critic's value at the end.
  • A3C is the asynchronous version: many workers run in parallel threads, each with its own copy of the environment, and push their updates to a shared set of parameters at their own pace. A2C does the same synchronously, which is simpler and works better on GPUs.
  • PPO and TRPO keep the actor-critic structure but limit how far one update may move the policy, which makes training much more stable. PPO is what many practitioners reach for first.

The ideas to keep are the ones you built here: sample actions from a stochastic policy, weigh the log-probability gradient by an advantage, and learn a critic to make the advantage less noisy.

PPO: Proximal Policy Optimization

Plain policy-gradient training has an awkward property: after one big update the policy can become much worse, and because the next batch of experience comes from that worse policy, it may never recover. Trust-region methods (TRPO) prevent this by limiting how far the new policy may move from the old one, but they are complicated to implement. Proximal Policy Optimization (PPO) gets most of the benefit with a very simple trick: it clips the probability ratio.

Let r = ฯ€_new(a | s) / ฯ€_old(a | s) be the ratio between the probability that the updated policy gives to the action that was actually taken and the probability the old policy gave it. Let A be the advantage of that action (how much better than average it turned out, as estimated by the critic in an actor-critic method, described above). The PPO objective for one sample is:

TEXT
L = min( r ยท A ,  clip(r, 1 - ฮต, 1 + ฮต) ยท A )        (ฮต is typically 0.1 to 0.2; the objective is maximized)
  • If the action was good (A > 0), increasing its probability helps, but only up to r = 1 + ฮต. Beyond that the objective stops growing, so there is nothing to gain from a bigger step.
  • If the action was bad (A < 0), decreasing its probability helps, but only down to r = 1 - ฮต.

The clipping removes the incentive to move the policy far from the one that collected the data, so PPO can safely reuse the same batch for several epochs of mini-batch gradient steps, which makes it sample-efficient. A full PPO implementation adds a value-function loss, an entropy bonus (to keep exploring) and generalized advantage estimation. PPO is the default policy-gradient method in many libraries, and it is the algorithm behind much of the reinforcement learning from human feedback used to fine-tune language models.

The example below shows the clipped objective as a function of the ratio, for a good and a bad action, and runs a tiny bandit-style policy optimization with and without clipping to show how large the policy change per update becomes.

Python The PPO clipped objective, and why clipping limits the policy change

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a good action the objective stops increasing once the ratio passes 1.2, and for a bad action it stops decreasing below 0.8, so neither can be pushed further by the gradient. In the second experiment, both runs start from a uniform policy and take 50 gradient steps on the same batch. Without clipping the policy keeps moving toward whatever the batch says, and the final change is larger; with clipping the update halts when the ratios leave the trust band, and the change stays smaller. That small, safe step per batch is what makes PPO robust.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In the REINFORCE update, what happens to an action that was followed by a high return?

    The update multiplies the gradient of log pi(a | s) by the return. A positive, large return pushes the log-probability of that action up, so the action becomes more likely in that state.
  2. 2. What does a baseline do in policy gradient methods?

    A baseline that does not depend on the chosen action does not change the expected gradient. Subtracting it makes the individual gradient estimates much less noisy, as the variance printout showed.
  3. 3. In an actor-critic method, what is the critic's job?

    The critic learns V(s). The TD error r + gamma * V(s') - V(s) tells the actor whether the action was better or worse than expected, and it is available after every step.
  4. 4. Why can a policy-based method be a better choice than Q-learning for continuous actions?

    Q-learning must find the action with the largest Q-value, which is hard when the action set is continuous. A policy network can simply output the parameters of a distribution and sample from it.

Summary

  • Policy gradient methods change the parameters of the policy directly, using G ยท โˆ‡ log ฯ€(a | s): good outcomes make their actions more likely.
  • REINFORCE plays a full episode and updates every step with the discounted return; it is unbiased but noisy.
  • A baseline does not change the average gradient but reduces its variance; the state value is the natural baseline.
  • Actor-critic learns a policy (actor) and a value function (critic) together, and the TD error trains both after every step.
  • A2C, A3C and PPO scale the same idea with neural networks, parallel workers and limits on how far one update can move the policy.

Next: return to value-based learning in Q-Learning, SARSA and Model-Free Methods, or move on to Project: Fake News and SMS Spam Detection.