Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: Balancing CartPole from Scratch

Build the CartPole physics yourself, then train policies that balance the pole with random search and the cross-entropy method, using only NumPy and a simple linear policy.

CartPole is the "hello world" of continuous-state control: a cart on a track carries a pole hinged on top. Each step you push the cart left or right, and the episode ends when the pole tilts more than 12 degrees or the cart leaves the track. Reward is +1 for every step you stay up, so more steps survived means a higher score. The standard limit is 500 steps; we will use 200 so that everything runs quickly in the browser.

In this project you will write the physics yourself (the same equations as Gymnasium's CartPole, introduced in the Gymnasium lesson), then train a policy with two simple derivative-free methods: random search and the cross-entropy method (CEM). Neither needs gradients or a value function, and both work on CartPole because a tiny linear policy is enough.

The environment

The state has four numbers: cart position x, cart velocity x_dot, pole angle theta and pole angular velocity theta_dot. The two actions push the cart with a force of -10 or +10 newtons. The physics are the classic cart-pole equations, integrated with a small Euler step of 0.02 seconds.

Python CartPole physics and a random agent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The random agent lasts roughly 20 steps. Even a simple hand-written rule that reacts to the pole's angle and angular velocity does far better, which tells us a simple function of the state should be enough to balance it.

A linear policy

Instead of hand-writing the rule, let us parametrize it and search for good parameters. The policy is:

TEXT
action = 1 (push right)  if  w . state > 0,  else 0 (push left)

The weight vector w has just four numbers. Our job is to find a w whose policy survives all 200 steps. This is policy search: treat the average episode return as a black-box function of w and optimize it.

Sample random weight vectors, evaluate each over a few episodes and keep the best. CartPole's solution space is generous enough that this often works within a few dozen tries.

Python Random search for a linear policy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Random search found a policy that balances for 200 steps, or close to it. Look at the signs of the weights: the ones for theta and theta_dot are positive, because pushing right when the pole leans right (and spinning right) is exactly what catches it.

Method 2: the cross-entropy method

Random search wastes the knowledge in its failures. The cross-entropy method learns a distribution over weight vectors. Each iteration:

  1. Sample a population of weight vectors from a Gaussian N(mean, std^2).
  2. Evaluate every candidate.
  3. Keep the best few (the elite, for example the top 20 percent).
  4. Refit the Gaussian to the elite: new mean and standard deviation.

The distribution shifts toward the region where good policies live and narrows as it finds them. Because it only uses rankings of returns, it is insensitive to reward scale.

flowchart LR
    G["Gaussian over weights"] --> S["Sample population"]
    S --> E["Evaluate each in the environment"]
    E --> K["Keep the elite"]
    K --> F["Refit mean and std"]
    F --> G

CEM is evolution in miniature: sample, select, refit.

Python Cross-entropy method on CartPole

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see the population's average score climb as the distribution tightens around good weights. CEM is a strong baseline for small policies and a useful sanity check for any RL implementation: if a deep RL agent cannot beat CEM on a simple control task, something is wrong.

Why not just use gradients?

Derivative-free methods work well when the policy has few parameters and episodes are cheap. They scale poorly to networks with millions of weights, where gradient-based policy gradients or PPO are far more sample-efficient. Still, evolution strategies (the cousin of CEM that estimates a gradient from random perturbations) have been used to train large networks when parallel simulation is cheap.

Extension ideas

  • Replace the linear policy by a tiny one-hidden-layer network and use the same CEM loop.
  • Make the task harder: shorter pole, heavier cart, observation noise, or a delayed action.
  • Report the standard deviation of the final scores over 10 independent CEM runs. How reliable is the method?
  • Port the environment to the Gymnasium API (subclass gym.Env) and verify it against the built-in CartPole.
  • Try REINFORCE with a linear softmax policy using the same environment.

Common mistakes

  • Evaluating each candidate on only one episode: lucky starts make bad policies look good. Average a few episodes.
  • Letting the standard deviation collapse to zero too early, which stops the search; add a floor.
  • Testing on the same random seeds used for training; always check on fresh starts.
  • Forgetting the time limit: a policy that survives 200 steps may simply be capped, not perfect.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In CartPole, what does the reward of +1 per step encourage?

    The episode ends when the pole falls or the cart leaves the track, so staying alive longer means a higher return.
  2. 2. What does the cross-entropy method do in each iteration?

    It is a sample, select and refit loop that uses only the ranking of returns.
  3. 3. Why average several episodes when scoring a candidate policy?

    Random initial states make single-episode returns unreliable; averaging reduces noise.
  4. 4. Which weights in a linear CartPole policy should end up positive?

    Pushing toward the side the pole leans (and rotates toward) catches it.

Summary

  • CartPole is a four-number state, two-action control problem; the physics fit in a few lines.
  • A linear policy action = (w . state > 0) is enough to balance the pole.
  • Random search finds a good w by sampling; the cross-entropy method learns a distribution over w and tightens it around the elite candidates.
  • Derivative-free search is a robust baseline, but gradient methods scale to large networks.
  • Always test on fresh random starts and average several episodes per candidate.

Next: Practice: Debugging RL and Self-Test