Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 37 ยท Power and Sample Size

Sample Size for Conversion Rates

Calculate the sample size for a conversion-rate A/B test. Learn the formula, use statsmodels, and see how baseline, minimum detectable effect, alpha and power change the answer.

Before launching a test you must answer: how many users per arm? Too few and you cannot detect the effect you care about; too many and you waste time and expose users to a worse variant longer than necessary. For conversion-rate metrics there is a clean formula, and Python can do it for you in a couple of lines.

The inputs

A sample-size calculation needs four choices:

  1. Baseline rate p_A: the control's current conversion rate, taken from recent data.
  2. Minimum detectable effect (MDE): the smallest change worth detecting, absolute (+1 point) or relative (+10%). This is a business decision, not a statistical one: below this size, shipping the change would not be worth it.
  3. Significance level alpha: typically 0.05 (two-sided).
  4. Power 1 - beta: typically 0.8 or 0.9.

The output is n per arm. Total users are 2n for a 50/50 split.

The formula

For a two-sided two-proportion z-test with equal arms:

n = (z_{1-alpha/2} * sqrt(2 * pbar * (1 - pbar)) + z_{power} * sqrt(p_A(1-p_A) + p_B(1-p_B)))^2 / (p_B - p_A)^2

where p_B = p_A + MDE and pbar = (p_A + p_B) / 2. You do not need to memorize it; notice that the effect appears squared in the denominator.

Python Sample size from the formula

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

About fifteen thousand users per arm are needed to detect a one-point lift on a 10% baseline with 80% power. If your site gets 2,000 visitors a day to this page, that is roughly two weeks.

Using statsmodels

In practice, use a library to avoid algebra mistakes. statsmodels computes power via Cohen's h effect size, which is based on an arcsine transform of the rates. Its answer agrees closely with the formula above.

Python statsmodels power analysis

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You can solve for any one of the four quantities (sample size, effect, alpha, power) by leaving it out. That makes it easy to answer "what is the smallest effect we can detect with the traffic we have?"

How the inputs change the answer

Python Sensitivity of the sample size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the patterns:

  • Smaller MDE costs a lot. Detecting +2% relative needs about 25 times the users of +10%.
  • Low baselines are expensive. A 2% conversion rate needs several times more users than 10% for the same relative lift, because conversions are rarer.
  • Stricter standards cost more. Raising power from 80% to 90%, or lowering alpha, increases the sample modestly.

Choosing the MDE honestly

The MDE is not "what we hope the change does." It is the smallest effect that would change your decision. Ask: "if the true lift were this small, would we still ship it?" If the answer is no, your MDE is too small and you are paying for precision you do not need. If you cannot afford the sample for your MDE, you have three honest options: accept a larger MDE, wait for more traffic, or test a bolder change.

flowchart TD
    A["Pick baseline from data"] --> B["Choose MDE from business value"]
    B --> C["Set alpha and power"]
    C --> D["Compute n per arm"]
    D --> E{"Traffic allows it<br/>in a few weeks?"}
    E -- yes --> F["Run the test"]
    E -- no --> G["Raise MDE, bolder change,<br/>or reduce variance"]
    G --> B

The calculation is a loop: if the answer is unaffordable, change the plan, not the rules.

Unequal splits and multiple variants

A 90/10 split needs more total users than 50/50 for the same power: with k as the ratio of arm sizes, solve_power(..., ratio=k) handles it. Equal splits are the most efficient for a fixed total. With several treatments compared against control, divide alpha across comparisons (Multiple Comparisons) and size accordingly.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What should the minimum detectable effect (MDE) be based on?

    The MDE is a business threshold: below it, the change would not be worth shipping, so there is no need to detect it.
  2. 2. What happens to the required sample size if you want to detect an effect that is 10 times smaller?

    Sample size scales roughly with 1 / effect squared, so a 10x smaller effect needs about 100x the users.
  3. 3. Why is a 2% baseline conversion rate more expensive to test than a 10% baseline for the same relative lift?

    The same relative lift corresponds to a smaller absolute difference and fewer events per user, so noise dominates more.
  4. 4. Which split is most efficient for a given total number of users?

    Equal arms minimize the variance of the difference for a fixed total sample.

Summary

  • Sample size depends on the baseline, the MDE, alpha and power; compute it before launch.
  • Use statsmodels (NormalIndPower with proportion_effectsize) or the closed formula.
  • Required users scale with 1 / MDE squared and grow when baselines are low.
  • Choose the MDE from the smallest effect that would change the decision.

Next: Sample Size for Means and the MDE