Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 37 ยท Getting Started

Simulating Experiments in Python

Build a reusable A/B test simulator in Python to generate fake experiment data with a known effect, split users at random, and understand the A/A test.

Throughout this course we generate our own experiment data. That is deliberate: with simulated data you know the true effect, so you can check whether a method gets it right, how often it is wrong, and what happens when you break the rules. Real experiments never give you that luxury, which is why simulation is how statisticians learn to trust their tools.

In this lesson you build the small simulator that later lessons reuse, learn to split users at random, and run the most important sanity check in experimentation: the A/A test.

A user table and a random split

We start with a table of users and assign each to A or B with a fair coin. Using NumPy's default_rng with a seed makes runs reproducible.

Python Random assignment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both arms end up with nearly identical country mixes and mobile shares even though we never balanced them on purpose. That is the magic of randomization: it balances the traits you know about and the ones you do not.

A reusable simulator

Now wrap the data generation in a function. It takes the baseline rate and the true lift and returns a table of outcomes. Later lessons call it again and again.

Python A conversion experiment simulator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The function returns one row per user, the same shape you get from a real experiment log. Everything you learn to do with this table (a z-test, a confidence interval, a power calculation) works the same on real data.

The A/A test: your most important sanity check

An A/A test gives both arms the same experience. Since there is no true difference, a well-behaved analysis should declare a significant result only about 5% of the time at alpha = 0.05. If it does so much more often, your assignment, logging or statistics are broken.

We will use a proportion z-test, derived properly in the next chapters; for now treat it as a function that returns a p-value. Run 2,000 A/A experiments and count the false alarms.

Python A/A test: how often do we cry wolf?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see a rate close to 5%. That is not a bug; it is the definition of alpha. Even when nothing changes, 1 in 20 tests will look "significant" by chance. This is also why you must decide on your analysis in advance: every extra look or extra metric is another ticket in that lottery.

Seeing the sampling distribution

A picture makes the noise concrete. We simulate 3,000 experiments with a true lift of one point and plot the histogram of observed lifts. The bell shape (the sampling distribution) is the foundation of every confidence interval in this course.

Python Distribution of observed lift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The standard deviation of this distribution is called the standard error of the lift. Everything in hypothesis testing is about comparing an observed lift with this standard error.

Tips for good simulations

  • Seed your generator (np.random.default_rng(seed)) so results are reproducible.
  • Know the truth, then break it. Simulate a world with no effect (A/A), a small effect, and a big one, and check your method in each.
  • Repeat many times. One simulated experiment shows noise; thousands show the behavior of the method.
  • Keep the simulator honest. Real data have skew, seasonality, bots and duplicate users. Add those when you want to test robustness.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is an A/A test?

    Because there is no true difference, an A/A test should produce false positives only at the rate alpha. Higher rates reveal bugs in assignment, logging or analysis.
  2. 2. In an A/A test at alpha = 0.05, about what share of tests are expected to be flagged significant?

    Alpha is the accepted false-positive rate, so about 1 in 20 A/A tests looks significant by chance.
  3. 3. Why do both arms end up with a similar country mix after random assignment, even if we never balanced it?

    Random assignment is independent of user traits, so in large samples every trait is distributed alike in both arms.
  4. 4. What does the standard deviation of the simulated lifts represent?

    The spread of the lift across repeated experiments is the standard error, the yardstick against which an observed lift is judged.

Summary

  • Simulation with a seeded numpy generator lets you test methods against a known truth.
  • Random assignment balances user traits across arms.
  • An A/A test should flag about alpha (5%) of runs as significant; much more means something is broken.
  • The spread of observed lifts across repetitions is the standard error.

Next: Writing Testable Hypotheses