Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 37 ยท Policy Gradients and Actor-Critic

The Policy Gradient Theorem and REINFORCE

Learn policy gradient methods: parametrized policies, softmax and Gaussian policies, the log-derivative trick, the policy gradient theorem and a runnable REINFORCE agent in NumPy.

Value-based methods such as Q-learning learn how good each action is and then pick the best one. The policy is a by-product: argmax of the values. Policy gradient methods skip the middleman. They represent the policy directly as a function with parameters theta, pi(a | s; theta), and adjust theta by gradient ascent to make good behavior more likely.

Why would you want that? Policies can be stochastic (needed for rock-paper-scissors or any hidden-information game), they handle continuous actions naturally (output the mean of a Gaussian instead of searching a max over infinitely many actions), and small parameter changes change behavior smoothly, which gives better convergence properties. This lesson derives the core result and implements the simplest algorithm, REINFORCE.

Parametrized policies

For discrete actions the standard choice is a softmax over preferences:

TEXT
pi(a | s) = exp(h(s, a)) / sum_b exp(h(s, b))

where h(s, a) is a table entry, a linear function of features, or the output of a neural network. For continuous actions, the policy is a Gaussian: the network outputs a mean mu(s) (and a standard deviation, learned or fixed), and the action is sampled from Normal(mu, sigma).

The goal is to maximize the expected return J(theta) = E[G_0] under the policy. We want grad J. The difficulty: theta affects which states are visited and which rewards are earned, and we do not know how the environment's dynamics depend on theta.

The log-derivative trick

Here is the key identity. For any function f and distribution p_theta(x):

TEXT
grad E[f(x)] = E[ f(x) * grad log p_theta(x) ]

because grad p = p * grad log p. The expectation of f times a score function can be estimated by sampling: you never need the derivative of f itself, and never need to differentiate the environment. Let us verify this numerically on a three-armed bandit with a softmax policy, where we can also compute the exact gradient.

Python The log-derivative trick on a 3-armed bandit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two gradients agree up to sampling noise. For a softmax policy the score function is grad_theta log pi(a) = onehot(a) - pi: raise the preference of the action taken, lower the others in proportion to their probability.

The policy gradient theorem

For a full episodic MDP the same idea gives the policy gradient theorem:

TEXT
grad J(theta) = E_pi [ sum_t  grad log pi(a_t | s_t; theta) * G_t ]

where G_t is the return from time t onward. In words: for each step, push up the log-probability of the action taken, in proportion to how much return followed it. No model of the environment appears anywhere. This theorem is the foundation of REINFORCE, actor-critic, PPO and almost everything in the rest of this chapter.

REINFORCE

REINFORCE is Monte Carlo policy gradient:

  1. Run an episode with the current policy, recording (s_t, a_t, r_t).
  2. Compute the return G_t for every step.
  3. Update theta <- theta + alpha * gamma^t * G_t * grad log pi(a_t | s_t) for each step.

For a tabular softmax policy with parameters theta[s, a], the gradient of the log-probability is simply onehot(a) - pi(. | s) for row s.

flowchart LR
    P["Policy pi(a | s; theta)"] --> E["Run an episode"]
    E --> G["Compute returns G_t"]
    G --> U["theta += alpha * G_t * grad log pi"]
    U --> P

REINFORCE: sample, score, nudge. Good episodes make their actions more likely.

The environment below is a 4-by-4 gridworld: the goal gives +1, the pit -1, each step -0.04. The agent starts in the top-left corner.

Python REINFORCE on a gridworld

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The average return rises as the policy sharpens toward a route that avoids the pit. Run it again with a different seed: REINFORCE results vary a lot between runs. That variability is the algorithm's weakness, and the next lesson fixes it.

Continuous actions: the Gaussian policy

With a Gaussian policy, pi(a | s) = Normal(mu(s), sigma^2), the score function is

TEXT
grad_mu log pi(a | s) = (a - mu) / sigma^2

so the update moves mu toward actions that earned above-average return and away from those that earned less. You will use this in the continuous-control lessons and see how PPO generalizes it.

Advantages and limits

Advantages. Learns stochastic policies; handles continuous actions; smooth policy changes; works when the optimal policy is simpler than the value function.

Limits. High variance; sample-inefficient (each episode is used once, then thrown away); can converge to a local optimum; step size matters a great deal, because one bad update can destroy the policy.

Common mistakes

  • Using the total episode return instead of the return from step t (the reward-to-go), which adds needless noise.
  • Forgetting that the gradient is ascent, not descent (sign errors make the agent get worse).
  • Not normalizing returns, so a few large episodes dominate.
  • Softmax overflow: always subtract the maximum before exp.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the log-derivative trick allow us to do?

    grad E[f] = E[f * grad log p], which is an expectation we can estimate from samples.
  2. 2. For a softmax policy, what is grad of log pi(a) with respect to the preferences?

    The preference of the taken action rises and the others fall in proportion to their probabilities.
  3. 3. In REINFORCE, how is each step's update weighted?

    Actions followed by a large return are made more likely; those followed by a small or negative return less.
  4. 4. Which is a known weakness of REINFORCE?

    Whole-episode Monte Carlo returns are noisy, so learning is slow and uneven; baselines and critics reduce this.

Summary

  • Policy gradient methods optimize a parametrized policy directly by gradient ascent on expected return.
  • The log-derivative trick turns the gradient into an expectation of return * grad log pi, estimable from samples.
  • The policy gradient theorem says each step's log-probability gradient is weighted by the return that followed.
  • REINFORCE is the Monte Carlo version: simple, unbiased, but noisy.
  • Gaussian policies extend the idea to continuous actions.

Next: Baselines and Variance Reduction