Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 37 ยท Power and Sample Size

Power Analysis by Simulation

Estimate A/B test power by simulation when formulas fall short. Build a reusable power simulator, handle skewed metrics and compare tests, with runnable Python.

Closed-form power formulas work for the textbook cases: a z-test on proportions, a t-test on means with known variance. Real experiments are messier: skewed revenue, capped metrics, bootstrap analyses, ratio metrics, clustered users. For these there is a universal method that needs no formula: simulate the experiment many times and count how often it succeeds.

The recipe

Power is the probability that the test rejects the null when the effect is real. So:

  1. Write a function that generates one experiment's data with a known true effect.
  2. Apply your actual analysis (the one you will run on real data) and record whether it declared significance.
  3. Repeat thousands of times. The share of significant runs is the power.
  4. Run the same simulation with no effect to check the false-positive rate is about alpha.
flowchart LR
    G["generate data<br/>with known effect"] --> T["run the planned test"]
    T --> R["record: significant?"]
    R --> L{"repeat N times"}
    L --> G
    L --> P["power = share significant"]

Because the analysis is the real one, the power estimate includes every quirk of your method.

Power for the two-proportion test

Start with a case where the formula exists, so you can see the simulation agree with it.

Python Simulated vs formula power

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two powers agree within simulation noise, and the no-effect run confirms the analysis stays near 5%. Now you can trust the simulator and extend it.

Skewed revenue

Here is where simulation earns its keep. We want the power to detect a +6% relative lift in revenue per user using Welch's t-test, with a realistic distribution: many zeros, a heavy lognormal tail. No tidy formula covers that exactly.

Python Power for a skewed revenue metric

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even at tens of thousands of users per arm the power for a 6% revenue lift is modest. That tells you before launch that either the MDE must be larger, the test longer, or variance must be reduced.

Comparing analysis choices

Simulation lets you ask "which analysis is better for my data?" Compare plain Welch against Welch on capped revenue with the same samples:

Python Does capping raise power?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Capping gives a small gain in power here (a little more with heavier tails), at the price of testing a slightly different quantity. Simulation quantifies that trade so you can decide with numbers.

Practical tips

  • Use enough runs. The standard error of an estimated power is sqrt(power * (1 - power) / runs); with 1,000 runs expect about +/- 1.5 points. More runs for a sharper answer.
  • Always include a no-effect run to check the false-positive rate of your actual analysis pipeline.
  • Search for the sample size. Run the simulation over a grid of n values and pick the smallest reaching your target power; bisection speeds it up.
  • Seed the generator so results reproduce.
  • Model reality. Use historical data: resample real user rows (bootstrap them) and add a synthetic effect, which keeps the true shape of the metric.
Python Find n for 80% power by search

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The search lands close to the formula-based answer from the earlier lesson, which is the point: both routes agree when the formula applies, and only simulation works when it does not.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the power estimate in a simulation?

    Power is the probability of detecting a real effect, so simulate with a known effect and count the successes.
  2. 2. Why run a simulation with no effect as well?

    A no-effect run validates the analysis pipeline: if it flags far more than 5%, the test is invalid.
  3. 3. When is power-by-simulation most valuable compared with a formula?

    Simulation uses your actual analysis and data shape, so it handles cases formulas do not cover.
  4. 4. Roughly how large is the standard error of a power estimate from 1,000 runs near 80% power?

    sqrt(0.8 * 0.2 / 1000) is about 0.0126, so around 1.3 points; use more runs to tighten it.

Summary

  • Estimate power by generating data with a known effect, running the real analysis and counting significant results.
  • Always add a no-effect simulation to verify the false-positive rate.
  • Simulation handles skewed metrics, capping, bootstrap tests and other cases with no formula, and lets you compare analysis choices.
  • Use enough runs and search over sample sizes to find the one that reaches your target power.

Next: The Peeking Problem