Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Policy Gradients and Actor-Critic

Actor-Critic and A2C

Learn actor-critic methods: a policy (actor) guided by a value function (critic), the TD error as advantage, A2C and A3C, generalized advantage estimation, with runnable NumPy code.

REINFORCE needs a complete episode before it can learn anything, and its Monte Carlo return is noisy. TD learning, from the TD lesson, solved the same two problems for value estimation: update after every step, and replace the long noisy return with a one-step bootstrapped estimate. Actor-critic applies that idea to policy gradients.

It has two parts that learn together. The actor is the policy pi(a | s; theta); it decides what to do. The critic is a value function V(s; w); it judges how good the situation is. The critic's judgment tells the actor which of its actions were better than expected.

The one-step actor-critic

After each transition (s, a, r, s'):

TEXT
delta = r + gamma * V(s') - V(s)           # TD error (the critic's surprise)
w     += beta  * delta * grad V(s)          # critic: reduce the surprise
theta += alpha * delta * grad log pi(a | s) # actor: repeat actions that surprised us positively

The TD error delta is a sample of the advantage: it is positive when the action turned out better than the critic expected and negative when worse. This replaces G_t - V(s) of REINFORCE with a one-step estimate, trading some bias (the critic may be wrong) for much lower variance, and it allows learning online, every step.

flowchart LR
    S["state s"] --> A["Actor pi(a | s)"]
    A -->|"action a"| ENV["Environment"]
    ENV -->|"reward r, next state s'"| C["Critic V(s)"]
    C -->|"TD error delta"| A
    C -->|"TD error delta"| C

The critic watches outcomes and computes the TD error; the actor and the critic both learn from it.

An actor-critic agent

Run this on the 4-by-4 gridworld (goal +1, pit -1, step -0.04). Both the actor (softmax preferences per state) and the critic (one value per state) are tables, so you can read what they learned.

Python One-step actor-critic on a gridworld

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The critic's values should increase toward the goal and be low next to the pit, while the actor's arrows follow the values around the pit. Both tables improved from the same stream of TD errors.

A2C and A3C

Deep versions use neural networks for the actor and critic, usually sharing early layers, with these refinements:

  • Advantage Actor-Critic (A2C) collects short rollouts (say 5 to 20 steps) from several environments run in parallel, computes an n-step advantage for every step, then does one synchronized gradient update. Parallel environments decorrelate the data without a replay buffer.
  • A3C (asynchronous) did the same with many workers updating shared weights without waiting for each other. A2C is simpler and usually just as good on a single GPU.
  • An entropy bonus is added to the actor's objective to keep the policy from becoming deterministic too early:
TEXT
loss = -log pi(a | s) * advantage  +  c_v * (return - V(s))^2  -  c_e * entropy(pi(. | s))

with a value-loss weight c_v around 0.5 and an entropy weight c_e around 0.01.

N-step returns and GAE

One-step TD has low variance and high bias; Monte Carlo is the reverse. N-step returns sit in between. Generalized Advantage Estimation (GAE) blends all n-step advantages with an exponentially decaying weight controlled by lambda:

TEXT
delta_t = r_t + gamma * V(s_(t+1)) - V(s_t)
A_t     = delta_t + (gamma * lambda) * delta_(t+1) + (gamma * lambda)^2 * delta_(t+2) + ...

lambda = 0 is the one-step TD advantage; lambda = 1 is the Monte Carlo advantage G_t - V(s_t). Values around 0.95 are a popular default. It can be computed in one backward pass:

Python Generalized advantage estimation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The lambda = 1 row matches the direct computation G_t - V(s_t), confirming the formula. Notice how the one-step advantage only sees the surprise at the final step while a larger lambda credits earlier steps with some of it.

Comparing the families

Method Learns Target Bias Variance Update
REINFORCE policy Monte Carlo return none high per episode
REINFORCE + baseline policy + V G_t - V none medium per episode
Actor-critic policy + V TD error some (critic error) low per step
A2C with GAE policy + V A_t(gamma, lambda) tunable tunable per rollout

Common mistakes

  • Letting the critic learn too slowly: a stale critic gives the actor misleading advantages. Many implementations give the critic a higher learning rate.
  • Forgetting the terminal condition: the target must not bootstrap beyond a terminal state.
  • Letting gradients from the value loss swamp the policy loss when they share a network; weight them with c_v.
  • Using one environment instance at a time in A2C; without parallelism the data are highly correlated.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In one-step actor-critic, what plays the role of the advantage?

    The TD error is a one-step sample of how much better the action was than the critic expected.
  2. 2. What is the benefit of actor-critic over REINFORCE?

    Bootstrapping with a critic replaces the noisy Monte Carlo return, at the cost of some bias from critic errors.
  3. 3. What do the values lambda = 0 and lambda = 1 give in GAE?

    Lambda controls the bias-variance trade-off between the one-step TD advantage and the full Monte Carlo advantage.
  4. 4. Why add an entropy bonus to the actor's loss?

    Rewarding high entropy keeps action probabilities spread out while the agent is still learning.

Summary

  • Actor-critic couples a policy (actor) with a value function (critic) trained from the same TD error.
  • The TD error is a low-variance, slightly biased advantage estimate that enables per-step updates.
  • A2C runs parallel environments, uses n-step advantages and adds a value loss and an entropy bonus.
  • GAE smoothly trades bias and variance with lambda.
  • The next lesson makes updates safer with trust regions and PPO.

Next: PPO and Trust Regions