Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Policy Gradients and Actor-Critic

PPO and Trust Regions

Learn why large policy updates collapse performance and how TRPO and PPO fix it: the surrogate objective, the clipped ratio, multiple epochs per batch and a runnable NumPy PPO agent.

Policy gradient methods share a danger: one bad update can ruin the policy, and since the next batch of data comes from that ruined policy, recovery may be slow or impossible. In supervised learning a bad step just wastes a little time; in RL it changes the data you collect. The fix is to take steps that are provably or heuristically small in policy space: the new policy must not behave too differently from the old one.

TRPO (Trust Region Policy Optimization) enforces that with a hard constraint. PPO (Proximal Policy Optimization) gets almost the same effect with a simple clipping trick that needs only a first-order optimizer. PPO is the default workhorse of modern deep RL, used from robotics to language-model fine-tuning.

Reusing data: the surrogate objective

Vanilla policy gradient uses each batch once, then discards it. To reuse a batch for several gradient steps, the data must be corrected for the fact that the policy changed after the first step. Importance sampling does that. Define the probability ratio:

TEXT
r(theta) = pi_theta(a | s) / pi_theta_old(a | s)

The surrogate objective is

TEXT
L(theta) = E[ r(theta) * A ]

where A is the advantage estimated with the old policy. At theta = theta_old the ratio is exactly 1 and the gradient equals the ordinary policy gradient. As theta moves away, the surrogate is only a trustworthy approximation of the real objective near theta_old. Maximize it too aggressively and the approximation breaks.

TRPO in one paragraph

TRPO maximizes the surrogate subject to a constraint on the average KL divergence between the old and new policies, KL(pi_old || pi_theta) <= delta. It solves this with a natural-gradient step (conjugate gradient) and a line search. It has strong guarantees but is complicated to implement and does not mix well with shared networks or dropout. PPO keeps the spirit and drops the machinery.

PPO's clipped objective

PPO limits how far the ratio can usefully move:

TEXT
L_clip = E[ min( r * A,  clip(r, 1 - eps, 1 + eps) * A ) ]

with eps typically 0.1 to 0.2. Read it case by case:

  • If A > 0 (good action): increasing the ratio helps until r = 1 + eps; after that the objective is flat, so there is no incentive to push probability further up.
  • If A < 0 (bad action): decreasing the ratio helps until r = 1 - eps; beyond that, no incentive to push it lower.
  • The min makes the objective pessimistic: it never rewards overshooting, but it still fully penalizes a ratio that moves the wrong way.
Python The PPO clipped objective, case by case

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For A = +1 the objective grows with the ratio up to 1.2 and then stays at 1.2 (flat, no gradient). For A = -1 the objective is -ratio for ratios above 0.8, and it flattens at -0.8 below. For a negative advantage with a large ratio such as 2.0, the objective is -2.0: the min keeps the unclipped, more negative value, so the gradient still pushes that ratio back down.

The PPO algorithm

flowchart TD
    A["Collect rollouts with pi_old"] --> B["Compute advantages (GAE) and returns"]
    B --> C["K epochs of mini-batch SGD on L_clip + value loss + entropy bonus"]
    C --> D["pi_old = pi"]
    D --> A

Collect a batch, do several epochs of clipped updates on it, then throw it away and collect fresh data.

A typical deep-network configuration: 2,048 steps per rollout, GAE with gamma = 0.99 and lambda = 0.95, 10 epochs of mini-batches of 64, eps = 0.2, an entropy coefficient of 0.0 to 0.01, a learning rate around 3e-4 and advantage normalization per batch.

A tabular PPO agent

The same 4-by-4 gridworld, a tabular softmax actor and a tabular critic. Each iteration collects 12 episodes with the current policy, computes advantages G - V(s), then takes 4 epochs of full-batch updates. We compare PPO's clipped update with the same multi-epoch update without clipping, and print how far the policy moved (average KL) in each case.

Python PPO with and without clipping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The comparison shows the clipped version moving the policy by a smaller KL per iteration, which is the whole point: trust the old data only while the new policy stays close to the old one. On this tiny problem both versions can learn, so judge by the KL and the stability of the return curve; on large neural networks, the unclipped version often collapses.

Practical details

  • Clip the value loss too and use a separate learning rate or weight (c_v) if actor and critic share layers.
  • Early stopping on KL: stop the epochs for a batch when the average KL to the old policy exceeds a target (like 0.02).
  • Continuous actions: use a Gaussian policy; the ratio is computed from log-probabilities, exp(logp_new - logp_old), which is numerically safer than dividing probabilities.
  • Normalize observations and advantages, and clip gradient norms (0.5 is common).
  • PPO-penalty is a variant that adds beta * KL to the loss with adaptive beta; clipping won in practice because it is simpler.
  • Reinforcement learning from human feedback (RLHF) uses PPO on language models with a KL penalty to the original model, a further proof that "do not move too far" is a general principle.

Common mistakes

  • Computing the ratio with probabilities from the current batch's policy instead of the stored old log-probabilities.
  • Running too many epochs per batch; clipping limits but does not eliminate drift.
  • Letting the value network see rewards at a very different scale from the advantages; normalize or scale rewards.
  • Interpreting the clip range as a hard guarantee: ratios can still leave [1 - eps, 1 + eps] between mini-batches.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the PPO probability ratio r(theta) measure?

    r = pi_new(a | s) / pi_old(a | s); it is 1 when the policies agree on that action.
  2. 2. With A > 0 and a ratio of 1.5 (eps = 0.2), what is the gradient of the clipped objective with respect to the policy?

    Beyond 1 + eps the objective is flat for positive advantages, so there is no incentive to raise the probability further.
  3. 3. Why does PPO run several epochs on one batch of data?

    Importance ratios plus clipping let the same rollout support multiple gradient steps without leaving the trust region too far.
  4. 4. How does PPO differ from TRPO?

    Clipping gives much of TRPO's stability at a fraction of the implementation complexity.

Summary

  • Large policy updates can destroy performance and the data you collect next; updates must stay close to the old policy.
  • The surrogate objective with the ratio pi_new / pi_old lets one batch support several gradient steps.
  • TRPO constrains the KL divergence; PPO clips the ratio to [1 - eps, 1 + eps], a simple and robust substitute.
  • PPO combines GAE advantages, a value loss, an entropy bonus and mini-batch epochs.
  • Gaussian policies extend PPO to continuous control; the same recipe underlies RLHF.

Next: Gymnasium: The Standard RL Environment API