Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Policy Gradients and Actor-Critic

Baselines and Variance Reduction

Learn why REINFORCE is noisy and how a baseline fixes it: proof that it adds no bias, a variance experiment in NumPy, learned value baselines and advantage normalization.

The policy gradient from the previous lesson is correct on average, but any single estimate is very noisy. Imagine a game where every episode scores between 100 and 110. Every action then has a hugely positive return, so REINFORCE pushes every action taken up, only slightly more for the slightly better ones. The useful signal (110 beats 100) is a tiny wiggle riding on a giant constant.

A baseline removes the constant. Instead of asking "was the return large?", we ask "was the return larger than usual?" This lesson shows why that is legal (it does not change the expected gradient), measures how much it helps, and builds the version that uses a learned baseline.

The baseline idea

Subtract a number b(s) from the return before weighting the gradient:

TEXT
grad J = E[ sum_t grad log pi(a_t | s_t) * (G_t - b(s_t)) ]

The baseline may depend on the state but not on the action. Under that condition it adds no bias:

TEXT
E_a[ grad log pi(a | s) * b(s) ] = b(s) * sum_a grad pi(a | s) = b(s) * grad( sum_a pi(a | s) ) = b(s) * grad(1) = 0

The probabilities always sum to one, so their gradient is zero, and any state-only baseline drops out on average. What changes is the variance of each sample. The best simple choice is the state value, b(s) = V(s). Then G_t - V(s_t) estimates the advantage A(s, a): how much better this action was than the policy's average action from that state.

If the baseline depended on the action, the argument above would fail and the gradient would be biased. Be careful with any baseline that looks at a_t.

Measuring the variance

Take a 3-armed bandit whose rewards are large and noisy: arms pay about 100, 101 and 105. Below we draw many single-sample gradient estimates with and without a baseline equal to the average reward, then compare their mean (should match, if the baseline is unbiased) and their spread.

Python Variance of the gradient estimate, with and without a baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The baseline version matches the exact gradient closely, so subtracting the baseline did not bias it. The version without a baseline is off by more because of its noise: its per-sample standard deviation is more than fifty times larger. To get the same accuracy you would need hundreds of times more data. Variance, not bias, is the enemy of policy gradients.

Reward-to-go: another variance reducer

An action at time t cannot affect rewards that were already paid before t. The policy gradient theorem therefore uses the reward-to-go G_t = r_t + gamma * r_(t+1) + ... and not the whole-episode return. Including past rewards adds terms with zero expected value and extra noise. REINFORCE in the last lesson already used G_t.

A learned baseline

For real problems we do not know V(s), so we learn it with a second function approximator, trained by regression to the observed returns:

TEXT
w <- w + beta * (G_t - V(s_t; w)) * grad V(s_t; w)

Now there are two learners: the policy (the actor), updated with (G_t - V) as the weight, and the value function (baseline), updated toward G_t. In a training loop it looks like this:

Python
G = r_t + gamma * G                      # return from step t, computed backwards
adv = G - V[s_t]                         # advantage estimate
theta[s_t] += alpha * adv * grad_log     # policy (actor) update
V[s_t] += beta * (G - V[s_t])            # baseline update

Here is a measurement on the 4-by-4 gridworld (goal +1, pit -1, step -0.04). We run a random policy, learn a tabular baseline from the first 600 episodes, then compare the per-episode gradient on 600 fresh episodes with and without the baseline: the mean gradient should agree, the variance should not.

Python Gradient variance on a gridworld, with a learned baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two mean gradients should point the same way with similar size (differences are sampling noise), while the total variance with the baseline is clearly smaller. Smaller variance means each update is more trustworthy, so you can use bigger steps and need fewer episodes.

Advantage normalization

In practice (especially in deep RL) people also normalize advantages within each batch: subtract the batch mean and divide by the batch standard deviation. It keeps the gradient scale stable regardless of the reward scale, and it makes learning rates transferable between tasks. A tiny constant (1e-8) in the denominator avoids dividing by zero.

TEXT
adv = (adv - adv.mean()) / (adv.std() + 1e-8)

Choosing a baseline

Baseline Notes
Constant (mean return) Simple, helps a lot over none; ignores the state
Learned V(s) Standard; makes G_t - V an advantage estimate
Moving average Cheap stand-in for the mean return
Critic with bootstrapping Replaces G_t entirely: actor-critic

Common mistakes

  • Using a baseline that depends on the action; this introduces bias.
  • Updating the baseline before using it in the policy update on the same step, leaking information.
  • Giving the baseline network a much higher or lower learning rate than needed, so the "advantage" is noise.
  • Normalizing advantages over a single episode of one step, where the standard deviation is zero.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does subtracting a state-dependent baseline not bias the policy gradient?

    Summing grad pi(a | s) over actions gives the gradient of 1, which is zero, so the baseline term vanishes in expectation.
  2. 2. What does G_t - V(s_t) estimate?

    It measures how much better the outcome was than the typical value of the state.
  3. 3. What is the main effect of a baseline?

    The mean of the gradient stays the same, but individual samples are much less noisy.
  4. 4. Which baseline would introduce bias?

    Only baselines independent of the action drop out of the expectation.

Summary

  • Policy gradients are unbiased but noisy; variance is the main practical problem.
  • Subtracting a state-only baseline leaves the expected gradient unchanged and reduces variance, often drastically.
  • The natural baseline is V(s), which turns G_t - V(s_t) into an advantage estimate.
  • Use reward-to-go, learn the baseline with regression and normalize advantages in each batch.
  • Replacing the Monte Carlo return by a bootstrapped estimate gives actor-critic methods.

Next: Actor-Critic and A2C