Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 57 ยท Applied Statistics

A/B Testing

Design and analyze A/B tests: hypotheses, sample size and power, the two-proportion z-test, confidence intervals for lift, peeking, and common pitfalls.

An A/B test is a randomized controlled experiment run online: visitors are randomly sent to version A (control) or version B (variant), and you compare an outcome such as the conversion rate. Random assignment is what lets you say B caused a difference, unlike the observational data behind most correlations. The statistics are the ones you already know (hypothesis tests, confidence intervals, power); what is new is the discipline of designing the test before you look at results.

flowchart LR
  A["Hypothesis + one primary metric"] --> B["Power analysis: how many visitors?"]
  B --> C["Randomize users to A or B"]
  C --> D["Run until planned sample is reached"]
  D --> E["Test + confidence interval for the lift"]
  E --> F["Decide: ship, stop or iterate"]

Step 1: define the question

Pick one primary metric (for example sign-up conversion rate) and decide in advance what difference matters. The hypotheses for a two-sided test:

  • Hโ‚€: p_B = p_A (no difference)
  • Hโ‚: p_B โ‰  p_A

The minimum detectable effect (MDE) is the smallest lift worth caring about, say 12% to 13.5% (an absolute lift of 1.5 percentage points). Tiny effects need enormous samples; if the change is worth little, it may not be worth testing.

Step 2: sample size and power

Two error types control the design: ฮฑ (false positive rate, usually 0.05) and ฮฒ (false negative rate), where power = 1 โˆ’ ฮฒ (usually 0.8). For two proportions pโ‚ and pโ‚‚, the required sample size per group is approximately

n = (zโ‚โ‚‹ฮฑ/โ‚‚ + zโ‚โ‚‹ฮฒ)ยฒ ร— [pโ‚(1โˆ’pโ‚) + pโ‚‚(1โˆ’pโ‚‚)] / (pโ‚‚ โˆ’ pโ‚)ยฒ

Notice the squared difference in the denominator: halving the effect you want to detect quadruples the sample.

Python Sample size for an A/B test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 3: run the test properly

  • Randomize at the unit you analyze (usually the user, not the page view) so the same person always sees the same version.
  • Run the planned duration, covering whole weeks so weekday and weekend behavior is balanced.
  • Check for a sample ratio mismatch: if you planned 50/50 and got 52/48 with large counts, something broke the randomization and the results cannot be trusted.
  • Change nothing mid-test.

Step 4: analyze with a two-proportion z-test

With x_A conversions out of n_A and x_B out of n_B, the pooled proportion is pฬ‚ = (x_A + x_B)/(n_A + n_B) and

z = (pฬ‚_B โˆ’ pฬ‚_A) / โˆš(pฬ‚(1โˆ’pฬ‚)(1/n_A + 1/n_B))

Always add a confidence interval for the difference. A p-value says only whether some difference is plausible; the interval says how large it might be and whether it is practically important.

Python Analyze a finished A/B test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Peeking: the most common mistake

If you check the results every day and stop as soon as p < 0.05, your real false positive rate is far above 5%, because each look is another chance for noise to cross the line. The simulation below runs an A/A test (both versions identical, so any "win" is false) and compares a single planned look with stopping at the first significant peek.

Python How peeking inflates false positives

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The planned look stays near 5%, while daily peeking multiplies it several times. Remedies: fix the sample size in advance and only decide at the end, or use a sequential testing method designed for repeated looks (group sequential boundaries, alpha-spending, always-valid p-values). Bayesian dashboards can also be misused if the stopping rule is ignored.

Other pitfalls

  • Multiple metrics and segments: testing 20 metrics at ฮฑ = 0.05 yields about one false "win". Declare one primary metric; treat the rest as exploratory, or correct for multiple comparisons (Multiple Testing and Effect Sizes).
  • Statistical vs practical significance: with millions of visitors a 0.01 point lift is significant yet worthless. Judge the interval against the cost of the change.
  • Novelty and primacy effects: users react to anything new at first. Run long enough for behavior to stabilize.
  • Interference: if users affect each other (marketplaces, social networks), randomizing individuals can mislead; randomize clusters instead.
  • Non-proportion metrics such as revenue per user are skewed; use a bootstrap or a log transform (Bootstrap and Permutation Tests).
  • Winner's curse: the observed lift of a winning variant tends to overestimate the true lift; expect some regression to the mean after launch.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What makes an A/B test able to support a causal claim?

    Randomization balances all other factors, known and unknown, on average, so differences in outcomes can be attributed to the change.
  2. 2. You want to detect a smaller lift. What happens to the required sample size?

    The detectable difference appears squared in the denominator of the sample-size formula, so halving it needs about four times as many users.
  3. 3. Why is stopping a test the first time p < 0.05 appears a problem?

    Each interim check is another test; with repeated peeking the overall false positive rate can be several times the nominal 5%.
  4. 4. A test with 2 million users shows a significant lift of 0.02 percentage points. What should you do?

    Huge samples make tiny effects significant. Practical significance depends on the size of the effect relative to the cost and benefit.

Summary

  • An A/B test is a randomized experiment: state one primary metric and the minimum effect that matters.
  • Use a power analysis to set the sample size in advance; smaller effects need much larger samples.
  • Analyze with a two-proportion z-test (or chi-square) and always report a confidence interval for the lift.
  • Do not peek and stop early; use sequential methods if you must monitor.
  • Beware of multiple metrics, novelty effects, interference and tiny-but-significant results.

Next: Time Series Basics