Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 37 ยท Power and Sample Size

Sample Size for Means and the MDE

Compute the sample size for A/B tests on means such as revenue per user, solve for the minimum detectable effect given your traffic, and understand the role of variance.

Conversion rates have a tidy formula because their variance is tied to the rate itself (p(1-p)). For a continuous metric such as revenue per user or time on page, the variance is a separate fact about your data that you must measure. This lesson shows how to size tests on means, and how to turn the problem around: given the traffic you have, what is the smallest effect you can detect?

The formula for means

For a two-sided test on the difference in means with equal arms and standard deviation sigma in each:

n = 2 * sigma^2 * (z_{1-alpha/2} + z_{power})^2 / delta^2

where delta is the minimum difference in means you want to detect. Two things control the answer:

  • The ratio delta / sigma (the standardized effect size, or Cohen's d). The bigger it is, the smaller the sample.
  • sigma itself. Noisy metrics (revenue with whales) need far more users than stable ones.
Python Sample size for a mean

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The revenue metric needs over a hundred thousand users per arm because its standard deviation is huge relative to the effect. Cut the standard deviation to a quarter and the requirement drops to a sixteenth. Variance is the lever, and the next chapters on variance reduction pull it.

Estimate sigma from history

You almost never know sigma in advance, so estimate it from recent data for the same metric, population and time window. Use the control group's historical standard deviation, and be careful with the unit: compute it per randomization unit over the same duration as the planned test. Revenue per user over 14 days has a different standard deviation than revenue per user per day.

Python Estimate sigma from historical data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The coefficient of variation (sigma / mean) is a handy summary: values above 1 mean the metric is very noisy relative to its average and will need big samples.

Solving for the MDE

Often the sample size is fixed by your traffic and calendar, and the real question is what you can detect. Rearranging the formula gives the MDE for a given n:

delta_min = (z_{1-alpha/2} + z_{power}) * sigma * sqrt(2 / n)

Python What can we detect with the traffic we have?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how slowly the MDE shrinks: quadrupling the sample only halves it, because of the sqrt(n). This is why chasing tiny effects gets expensive so quickly.

flowchart LR
    T["Traffic per day"] --> N["n per arm in a fixed window"]
    S["sigma of the metric"] --> M["MDE"]
    N --> M
    A["alpha, power"] --> M
    M --> D{"Is the MDE below<br/>what matters?"}
    D -- no --> R["Reduce variance, run longer,<br/>or change a bolder thing"]
    D -- yes --> G["Good to launch"]

Start from what you can afford and read off what you can detect; compare it with what you need.

Ratio metrics and per-user averages

If the metric is a ratio of two per-user quantities (revenue per session), size the test using the standard deviation of the per-user linearized value (delta method) rather than treating sessions as independent. For binary metrics measured per user, the conversion formula applies. If you randomize users but each has many sessions, variance is driven by the number of users, not sessions.

Unequal allocation and attrition

  • Unequal split: a control share of r and treatment of 1-r needs n_total = n_equal / (4 r (1 - r)); at 90/10 that is about 2.8 times the 50/50 total.
  • Attrition and exposure: if only 60% of assigned users reach the changed page, effective sample size is 60% of assigned, and the effect on the assigned population is diluted. Plan with the triggered population, and inflate the target accordingly.
Python Dilution from partial exposure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Analyzing only users who actually reached the change (and logging exposure the same way in both arms) is far more efficient than diluting the effect across everyone assigned.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. For a mean metric, which two quantities determine the standardized effect size?

    Cohen's d is delta divided by sigma. A noisy metric (large sigma) makes the same delta much harder to detect.
  2. 2. If you quadruple the sample size, how does the detectable effect change?

    The MDE is proportional to 1 / sqrt(n), so four times the users halves the minimum detectable effect.
  3. 3. How should you estimate sigma for the sample-size calculation?

    Use per-unit historical variation over the same duration as the planned experiment.
  4. 4. Only 40% of assigned users reach the changed page. What is the efficient analysis?

    Restricting to the triggered population removes unaffected users who add noise without carrying signal.

Summary

  • For means, n = 2 * sigma^2 * (z_alpha + z_power)^2 / delta^2; variance is the key driver.
  • Estimate sigma from history; the coefficient of variation shows how noisy a metric is.
  • Solve for the MDE when traffic is fixed; it shrinks only with the square root of n.
  • Analyze the exposed population and plan for dilution and unequal splits.

Next: Power Analysis by Simulation