Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 76 ยท Statistics for Data Science

A/B Testing and Experiments

Design and analyze A/B tests in Python: randomization, metrics, power and sample size, the peeking problem and how to interpret lift, with live simulations.

An A/B test is a controlled experiment: you split users at random into two groups, show group A the current version and group B a change, and compare an outcome. It is the most reliable way to learn whether a change causes a difference, because randomization removes the other explanations. Companies run thousands of them, for button colors, ranking algorithms, prices and emails.

Running the test is easy. Running it so that the answer can be trusted is the craft: choosing the right metric, collecting enough data, not stopping early because the numbers look good, and reading the lift honestly. This lesson walks through the whole design with live simulations. It uses the tools from hypothesis testing and confidence intervals, and the idea behind it, why randomizing works, is the subject of causal thinking.

Why randomize?

Suppose you ship the new checkout page and compare this week's conversion with last week's. Maybe conversion rose because of the new page, or because it was payday, or because a marketing campaign started. Everything else changed too. With random assignment, the two groups are alike in every way, measured or not, except the treatment. Any difference beyond chance must come from the change.

flowchart LR
    U["Eligible users"] --> R{"Random<br/>assignment"}
    R --> A["Group A<br/>current page"]
    R --> B["Group B<br/>new page"]
    A --> MA["Measure outcome"]
    B --> MB["Measure outcome"]
    MA --> C["Compare:<br/>lift + uncertainty"]
    MB --> C

Random assignment makes the groups comparable, so a difference in outcomes can be attributed to the change.

Randomize the right unit. If users can visit on several days and devices, assign by user ID (a hash of the ID keeps each person in one group), not by page view; otherwise one person sees both versions and the observations are not independent.

Choosing the metric before the test

Write down three things before launching:

  • Primary metric: the one number that decides the test, such as purchase conversion rate. One primary metric keeps the false-alarm rate honest.
  • Guardrail metrics: things that must not get worse, such as page load time, refund rate or unsubscribes. A change that lifts clicks but doubles complaints is not a win.
  • Diagnostic metrics: secondary numbers that explain why the primary moved. They are for understanding, not for declaring victory.

A good metric is sensitive (it moves when the change matters), attributable to the change, and hard to game. Revenue per user is closer to business value than clicks but is noisier and needs more data. Many teams pick a proxy, such as add-to-cart rate, when the real goal is slow to measure.

Running and analyzing a simple test

For a yes-or-no outcome like conversion, compare two proportions. The next example simulates 8,000 visitors, assigns each to a group with a coin flip, and analyzes the result. Notice the sample ratio check at the end: if you planned a 50/50 split and the groups are noticeably uneven, something in the assignment or logging is broken, and you should not trust the result until you find out why.

Python Simulate a test and compare two conversion rates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In this simulation the true gain is 1.5 percentage points (10.0% to 11.5%), but the measured gain is 2.5 points. That is not a bug; it is sampling noise, and the interval (about 1.2 to 3.8 points) correctly contains the truth. It is also a warning. Tests that happen to cross the significance line tend to overstate the effect, a pattern called the winner's curse. Report the interval, and expect the real-world gain to be a little smaller than the headline.

Also distinguish absolute lift (percentage points) from relative lift (percent change). "Conversion rose 27%" and "conversion rose 2.5 points" describe the same result, but the first sounds bigger. Say which one you mean.

Power and sample size

Before running a test, ask: how many users do I need? The answer depends on four connected quantities:

  • the minimum detectable effect (the smallest change worth detecting),
  • the significance level alpha (usually 0.05),
  • the power (the chance of detecting the effect if it is real; 80% is a common target),
  • and the noise in the metric (for proportions, the baseline rate).

Fix any three and the fourth follows. Under-powered tests are one of the most common failures: with too little data a real improvement is missed, and the team wrongly concludes "it does nothing". statsmodels computes sample sizes, and a simulation can check them.

Python How many visitors do we need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

About 6,700 visitors per group are needed for a 15% relative lift on a 10% baseline, and the simulation confirms power near 80%. Notice the last lines: halving the lift you want to detect roughly quadruples the required traffic. A site with low traffic cannot detect small effects, and it is better to know that before starting. Options then are to test bolder changes, pick a more sensitive metric, or run longer.

Run the test for whole weeks, so that every weekday is represented equally, and decide the duration from the sample-size calculation, not from the results.

The peeking problem

It is tempting to check the dashboard every day and stop as soon as p drops below 0.05. This breaks the test. A p-value is only valid if you look once, at the planned end. Each additional look is another chance for noise to cross the line, so the real false-alarm rate grows far above 5%. The simulation below runs A/A tests, where both groups are identical, so every "significant" result is wrong by construction.

Python Peeking inflates false positives

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Looking once gives the promised 5% (4.5% here, within simulation noise). Peeking ten times and stopping at the first "win" gives about 18%, nearly one in five empty tests declared a success. If your organization needs to monitor continuously, use methods designed for it: sequential tests or group-sequential designs that spend alpha across the looks, or Bayesian approaches (see Bayesian thinking). Do not simply reuse a fixed-sample p-value.

Interpreting the result

Once the test finishes at its planned size:

  1. Check validity first: sample ratio, logging errors, bots, a novelty effect that fades after a week.
  2. Look at the primary metric: estimate, interval and p-value. Is the entire interval above the smallest effect you care about?
  3. Check the guardrails. A win on the primary metric with a loss on a guardrail needs a discussion, not a launch.
  4. Be careful with subgroups. Splitting a flat overall result into twelve segments will always find one that looks good; treat that as a hypothesis for a follow-up test (see multiple comparisons in the previous lesson).
  5. Make a decision, including a decision to do nothing. "No detectable difference at this sample size" is a valid and useful result: it says the change is not worth its cost.

Tip: Keep a simple log of every test: hypothesis, metric, planned sample size, dates, result and decision. Over time it protects you from repeating failed ideas and shows how often your hunches actually win (usually far less than half the time).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main reason to randomize users into A and B?

    Random assignment balances both measured and unmeasured factors in expectation, which is what allows a causal reading of the difference.
  2. 2. You want to detect a lift half as large as before, with the same alpha and power. What happens to the required sample size?

    Required sample size scales with the inverse square of the effect, so halving the effect needs about four times the data.
  3. 3. A team checks the dashboard daily and stops the test the first day p < 0.05. What is the problem?

    A fixed-sample p-value is valid only at the planned end. Repeated peeking inflates false positives; use sequential methods if you must monitor.
  4. 4. The two groups were planned as 50/50 but contain 5,200 and 4,800 users out of 10,000, and a ratio test gives a very small p-value. What should you do?

    A sample ratio mismatch means the assignment or data collection is broken or biased, so downstream comparisons cannot be trusted until the cause is found.

Summary

  • Random assignment is what turns a comparison into a causal measurement; randomize at the right unit, such as the user.
  • Fix the primary metric, guardrails and analysis plan before the test starts.
  • Plan the sample size from the minimum detectable effect, alpha and power; small effects need much more traffic.
  • Do not peek and stop early with a fixed-sample p-value; use the planned duration or a sequential method.
  • Report lift with an interval, in absolute and relative terms, and expect winners to be a little optimistic.
  • "No detectable difference" is a legitimate, useful outcome.

Next: Causal Thinking and Confounding, for questions where you cannot randomize.