Skip to content
Tutorials Center
My learning

Topics

Lesson 51 of 57 ยท Projects

Project: Analyzing an A/B Test

Analyze a full A/B test: check randomization, test conversion with a z-test, compare revenue with a bootstrap, add a Bayesian view and write a clear recommendation.

Your company redesigned its checkout page (version B). For two weeks, visitors were randomly shown the old page (A) or the new page (B). Your job: decide whether to ship B. You will check the experiment's health, analyze the primary metric (conversion), analyze a secondary metric (revenue per visitor) whose distribution is awkward, look at the result in a Bayesian way, and write the recommendation.

This project ties together ideas from A/B Testing, Bootstrap and Permutation Tests and Bayesian Inference with the Beta-Binomial Model. The data is simulated with a fixed seed. The pre-registered plan said: primary metric is conversion rate; planned sample 10,000 per group; significance level 0.05; secondary metric is revenue per visitor.

flowchart TD
  A["1 Sanity checks: sample ratio, balance"] --> B["2 Primary metric: conversion z-test + CI"]
  B --> C["3 Secondary metric: revenue per visitor, bootstrap"]
  C --> D["4 Bayesian view: P(B better)"]
  D --> E["5 Recommendation and caveats"]

Step 1: sanity checks

Before looking at any outcome, verify that the experiment ran correctly. A sample ratio mismatch (SRM) means the observed split differs from the planned 50/50 by more than chance allows, which usually signals a bug in assignment or logging. Test it with a chi-square goodness-of-fit test; use a strict threshold (such as 0.001) because any SRM invalidates the experiment.

Python Generate the experiment and check the split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The split passes the check. (Also check that device type, country and new-vs-returning visitors are balanced across groups; here we assume they are.)

Step 2: the primary metric

Run the two-proportion z-test and report the lift with a confidence interval. The pre-registered decision rule: ship only if the result is significant and the lower end of the interval is positive.

Python Test and estimate the conversion lift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 3: revenue per visitor

Revenue is awkward: most visitors spend nothing and the buyers' amounts are skewed, so a t-test's normality assumption is shaky, and the mean is dominated by a few big orders. A bootstrap gives an interval for the difference in mean revenue per visitor without assuming a shape.

Python Bootstrap the difference in revenue per visitor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Interpret carefully: if the interval for revenue comfortably includes 0, then the extra conversions were not clearly translated into extra revenue. That can happen when the new design attracts more low-value orders. This is why you should decide in advance which metric is primary and consider guardrail metrics too (refunds, support tickets, page speed).

Step 4: a Bayesian view

Stakeholders usually ask, "How likely is it that B is better?" A frequentist interval does not answer that directly; a posterior does. Using flat Beta(1, 1) priors:

Python Posterior probability that B beats A

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line is a business-friendly question: "How likely is the lift to be big enough to matter?" Choose a threshold from the cost of the redesign.

Step 5: the recommendation

A good write-up is short and complete:

Recommendation: ship B. Conversion rose from 10.1% to 11.4% (absolute lift 1.4 points; 95% CI 0.5 to 2.2 points; p โ‰ˆ 0.002). The split was healthy (no sample ratio mismatch). Revenue per visitor shows no clear change (interval includes zero), so monitor order value after launch. The posterior probability that B is better is above 99.9%. Caveats: two weeks of data may include novelty effects; only desktop and mobile web were tested; we ran one primary test with no peeking, so the p-value is not inflated.

Notice the shape: the decision first, then the evidence with numbers and intervals, then the limits. The exact numbers you saw in the runs above determine your own wording, so always phrase it from the actual output.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a sample ratio mismatch indicate?

    If a 50/50 design gives a significantly different split, the randomization or data collection is broken and results cannot be trusted.
  2. 2. Why use a bootstrap for revenue per visitor?

    The bootstrap resamples the data and needs no distribution assumption, which suits skewed, zero-inflated metrics.
  3. 3. Conversion rises significantly but revenue per visitor shows no clear change. What is a sensible response?

    Report the full picture. Searching for significance in another metric would be p-hacking.
  4. 4. What extra question can a Bayesian posterior answer easily?

    From posterior draws you can compute the probability of any event, such as the lift being above half a percentage point.

Summary

  • Verify the experiment first (sample ratio, balance), then analyze the pre-registered primary metric.
  • Report lifts with confidence intervals and a decision rule fixed in advance.
  • Use a bootstrap for skewed metrics like revenue; use guardrail metrics for hidden costs.
  • A Bayesian posterior gives direct probabilities such as P(B is better) and P(lift is large enough).
  • Write the recommendation first, then the evidence and the caveats.

Next: Project: Building a Regression Model