Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 38 ยท Projects

Project: Analyze an A/B Test

Analyze a simulated A/B test with scipy.stats. Run proportion tests, build bootstrap confidence intervals, estimate power and write a decision memo.

An A/B test is the simplest controlled experiment a product team can run. Half of the visitors see version A, the other half see version B, and you count how many of them convert (sign up, buy, click). The hard part is not the counting. It is deciding whether a small difference in two percentages is a real effect or just luck, and how much evidence you need before you act.

In this project you will play the analyst. You get conversion data from a test, run the classic proportion tests, put a bootstrap confidence interval around the lift, estimate how much traffic the test really needed, and finish with a short decision memo. It uses ideas from Hypothesis Testing and Correlation, Regression and Bootstrap, so skim those first if the vocabulary feels new.

flowchart LR
    A["Simulate or load<br/>conversion data"] --> B["Proportion tests<br/>(chi-square, z, Fisher)"]
    B --> C["Bootstrap interval<br/>for the lift"]
    C --> D["Power and<br/>sample size"]
    D --> E["Decision memo"]

The five steps of the analysis. Power comes after the test here, but in real work you plan it before the test starts.

The data

We have no real customers, so we simulate them. Version A converts at a true rate of 10.0 percent and version B at 11.5 percent. That is a 15 percent relative improvement, which most teams would be happy with. Each version gets 4,000 visitors. In real life you never know the true rates; you only see the counts. Simulating lets us check later whether our methods are honest.

Each visitor is one Bernoulli trial: rng.random(n) < p gives an array of True (converted) and False values. A fixed seed makes the run repeatable.

Step 1: tests for two proportions

The question is "do A and B convert at the same rate?". Three standard tools answer it:

  • The chi-square test of independence on the 2 by 2 table of converted / not converted.
  • A two-proportion z-test, which is the same calculation written as a z score. Without continuity correction, it gives exactly the same p-value as the chi-square test on a 2 by 2 table.
  • Fisher's exact test, which makes no large-sample approximation. It is a good cross-check when counts are small.
Python Simulate the test and compare the proportions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three p-values land around 0.17 to 0.18. With the usual threshold of 0.05 you cannot call B a winner. But notice what this does not say: it does not say B is the same as A. We simulated a real 1.5 point improvement, and the test missed it. A p-value above 0.05 means "not enough evidence", not "no effect".

The z-test uses a pooled proportion for the standard error because the null hypothesis says both versions share one rate. A confidence interval for the difference uses the two separate proportions instead, because there we are not assuming they are equal.

Step 2: how big could the lift be?

Decisions need a range, not only a yes or no. A confidence interval for the difference tells you which lifts are compatible with the data. You can build one with a formula (the Wald interval) or by resampling. scipy.stats.bootstrap resamples both groups with replacement thousands of times, computes the difference each time and reads the interval off the percentiles.

The statistic must accept the samples as separate arguments and an axis keyword, so that SciPy can compute thousands of resamples in one vectorized call.

Python Bootstrap confidence interval for the lift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both intervals run from about -0.4 to +2.3 percentage points. The interval includes zero, which matches the p-value, but most of it lies on the positive side. That is a more useful message than "not significant": B is probably not worse by much, and could be better by two points. The last line is a rough resampling probability, not a Bayesian probability of B being better; do not present it as one.

Step 3: power and sample size

The test failed to detect a real improvement. Was that bad luck or bad design? Power is the probability that a test detects an effect of a given size when it really exists. Conventional designs aim for 80 percent. We can estimate power by simulation: run the whole experiment thousands of times with known true rates and count how often the test is significant.

For two proportions you can also compute the needed sample size per group with a formula. With significance level $\alpha$ and power $1-\beta$:

n = ( z(1-alpha/2) * sqrt(2 * pbar * (1 - pbar)) + z(power) * sqrt(p1 * (1 - p1) + p2 * (1 - p2)) )^2 / (p2 - p1)^2

where pbar is the mean of the two rates and z(q) is the normal quantile stats.norm.ppf(q).

Python Simulated power and the sample-size formula

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 4,000 visitors per version the power is only about 58 percent, so this test was likely to miss the effect even though it was real. The formula says roughly 6,700 per version are needed for 80 percent. The last line is a calibration check: with no true difference the test should be significant about 5 percent of the time, and it is.

Always fix the minimum detectable effect and sample size before the test starts. Choosing them afterwards, based on the result, is one of the easiest ways to fool yourself.

Step 4: why you must not peek

A tempting habit is to check the dashboard every day and stop the moment the p-value drops below 0.05. Every extra look gives chance another opportunity to cross the line. The simulation below runs thousands of A/A tests, where both versions are identical, and checks ten times along the way.

Python The cost of peeking at an A/A test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One look at the planned end keeps the false-positive rate near the promised 5 percent. Stopping at the first significant look among ten roughly quadruples it. If you must monitor continuously, use a sequential method designed for it; do not reuse a fixed-sample p-value.

Step 5: the decision memo

Numbers do not make decisions, people do, and people read short memos. A good one states the result, the uncertainty and a recommendation in plain language. Build it from the computed values so that it never disagrees with the analysis.

Python Generate the report from the results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The recommendation is KEEP TESTING: the data are promising but inconclusive, and the test is too small to settle the question. Notice that the memo asks for about 16,500 visitors per version, far more than the 6,700 computed earlier. The memo plugs in the observed lift of 0.95 points, which is smaller than the 1.5 points we designed for, and small effects need many more visitors. That is a useful warning: if the true lift is only a point, a test this size would struggle to prove it. Writing that sentence honestly is half of the job. A neat "B wins!" built on an underpowered test is the usual way A/B programs lose credibility.

Extending the project

  • Add a revenue per visitor metric. It is heavily skewed, so use stats.bootstrap on the mean or stats.mannwhitneyu instead of a z-test.
  • Compare four variants and correct for multiple comparisons, as covered in Hypothesis Testing.
  • Replace the Wald interval with the Wilson interval from stats.binomtest(...).proportion_ci(method="wilson") for each arm.
  • Plot the bootstrap distribution of the lift with matplotlib and shade the 95 percent interval.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A test gives p = 0.17 for the difference between two conversion rates. What can you conclude?

    A large p-value means the data are compatible with no difference, not that you proved there is none. A real effect can easily hide in a small sample.
  2. 2. What does statistical power measure?

    Power is 1 minus the Type II error rate. It depends on effect size, sample size and the significance level.
  3. 3. Why does the bootstrap statistic function take an axis argument?

    With vectorized=True, bootstrap passes arrays holding all resamples stacked along an axis. A statistic written with mean(axis=axis) handles them in a single pass.
  4. 4. You look at an A/B test every day and stop as soon as p < 0.05. What happens to the false-positive rate?

    Each look is another chance for noise to cross the threshold, so the overall error rate inflates. The simulation in this lesson shows it climbing to roughly 19 percent with ten looks.

Summary

  • Compare two conversion rates with a chi-square test on the 2 by 2 table, a pooled z-test or Fisher's exact test; the first two agree exactly.
  • A non-significant result means "not enough evidence". Report an interval for the lift so readers see which effects remain plausible.
  • scipy.stats.bootstrap builds intervals for any statistic; write the statistic with an axis argument and pass vectorized=True for speed.
  • Estimate power by simulation or with the sample-size formula, and decide the sample size before the test starts.
  • Peeking at a running test and stopping early inflates false positives. Finish the planned sample or use a sequential design.
  • A decision memo states the result, the uncertainty and a recommendation, generated from the computed numbers.

Next project: simulate how heat spreads with sparse matrices in Project: Simulate Heat Diffusion with Sparse Solvers.