Skip to content
Tutorials Center
My learning

Topics

Lesson 65 of 76 ยท Projects

Project: Experiment Readout

Analyse a simulated A/B test end to end in Python: power plan, health checks, conversion test, confidence interval, guardrails, segments and a decision memo.

An experiment is only finished when someone has made a decision from it. The readout is the document and the analysis that get you there: was the test healthy, what is the effect and how sure are we, did anything get worse, and what should we do? In this project you will produce a complete readout for a simulated checkout-page test, following the plan you would have written before the test started.

The ideas come from A/B Testing and Experiments, Hypothesis Testing Done Right and Estimation and Confidence Intervals. If you have not met them yet, skim those first. The data is generated with a fixed seed, so every number below matches what you will see on screen. Because we simulate it, we also know the true effect, which lets you judge how well the analysis did.

flowchart LR
    A["1. Plan<br/>(hypothesis, metric,<br/>sample size)"] --> B["2. Health checks<br/>(sample ratio, balance)"]
    B --> C["3. Primary result<br/>(effect + interval)"]
    C --> D["4. Secondary and<br/>guardrail metrics"]
    D --> E["5. Segments<br/>(carefully)"]
    E --> F["6. Decision memo"]

Decide the plan first, check the test's health second, and only then look at the result.

Step 1: The plan, written before the test

The readout starts with what was decided before any data arrived. Writing it down prevents the most common failure of experiments: choosing the metric, the segment or the stopping time after seeing which one looks good.

  • Change: a redesigned checkout page (variant B) against the current page (control A). Users are assigned at random, 50/50.
  • Hypothesis: the new page raises the share of visitors who complete a purchase.
  • Primary metric: conversion rate, the share of visitors with a completed purchase during the test. This one metric decides the outcome.
  • Secondary metric: revenue per visitor (informative, not decisive).
  • Guardrail: the share of visitors who contact support within seven days. It must not get meaningfully worse.
  • Baseline and smallest effect worth detecting: the current rate is about 8.0%. A rise to 8.8% (10% relative) would justify the redesign's cost; smaller changes would not.
  • Error rates: 5% false positive rate (two-sided), 80% power.
  • Decision rule: ship if the primary lift is positive with a 95% interval that excludes zero, and the guardrail is not worse by more than 0.5 percentage points.

From these inputs we compute the sample size needed so that the test can detect the smallest effect that matters.

Python Step 1: how many visitors do we need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how quickly the requirement grows as the effect shrinks: halving the effect roughly quadruples the sample. The table says we need about 18,900 visitors per arm for the 8.0% to 8.8% change, and about 73,800 for half that effect. With 20,000 per arm the power is 82% for the effect we care about but only 31% for the smaller one, so a "not significant" result would not prove that a small effect is absent. The plan fixes the sample size in advance, so we will not peek at the result and stop early.

Step 2: Health checks before looking at the effect

Two things can ruin an experiment without any statistics: the randomization did not work, or the data is broken. Check both before you open the results.

  1. Sample ratio mismatch (SRM). If assignment is 50/50, the arm sizes should be close to equal. A significant departure (a chi-square test against the planned split) means users were lost or duplicated unevenly, and the comparison is no longer between like groups. Use a strict threshold such as p < 0.001 for SRM.
  2. Covariate balance. Characteristics fixed before the test, such as device type or whether the visitor is new, should look the same in both arms.
  3. Data hygiene. No visitor in both arms, no duplicate rows, dates inside the test window.
Python Step 2: sample ratio, balance and hygiene checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All checks pass: the arms are almost exactly equal in size, device and visitor type are balanced (p-values far from small), and the data has no duplicates or impossible values. If the SRM check had failed, the right response would be to investigate the assignment or logging code and not to report the effect.

One more point about the balance tests: with several covariates, an occasional small p-value is expected by chance. A balance test is a smoke alarm; a single weak alarm among many tests is not a fire.

Step 3: The primary result

Now the question the test was designed for. For conversion rates, report:

  • the two rates and the absolute difference (percentage points), which is what moves the business;
  • the relative lift, which is easier to remember;
  • a 95% confidence interval for the difference;
  • a p-value from a two-proportion z-test (a formality here, since the interval carries the same information and more).
Python Step 3: conversion rate, difference, interval and test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Variant B converted 8.48% of visitors against 7.66% for control: a difference of 0.82 percentage points (10.7% relative), with a 95% interval from 0.29 to 1.35 points and p = 0.003. Read the interval, not just the p-value. It states the range of effects consistent with the data: anything from a modest 0.3 points to a strong 1.35 points. The bootstrap interval for the relative lift, 3% to 18%, is just as wide, which is why you should never report a lift without its interval.

Resist two temptations. Do not say "the probability that B is better is 99.9%" from a p-value; that is a different (Bayesian) statement (see Bayesian Thinking). And do not call a small effect "huge" because the p-value is small; the p-value reflects the sample size as well as the effect.

Step 4: Secondary and guardrail metrics

The primary metric decides; the others protect us from nasty surprises and add context.

  • Revenue per visitor mixes a yes/no outcome with a skewed amount (most visitors spend nothing; purchases vary). A t-test is borderline for such data, so use a bootstrap interval for the difference in means.
  • The guardrail, support contacts, is tested for harm: we want to be confident that B is not worse by more than the 0.5-point margin from our plan. That is a non-inferiority check: look at the upper end of the interval for the increase.

Treat these as supporting evidence. Testing many metrics at the 5% level guarantees some false alarms, so a surprising result in a secondary metric is a reason for a follow-up test, not for a headline.

Python Step 4: revenue per visitor and the support guardrail

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Revenue per visitor rose from 3.34 to 3.76 (a difference of 0.42, interval 0.12 to 0.71). The average order value barely changed (43.60 against 44.33), so the extra revenue comes from more buyers, not from bigger baskets. The interval is wide relative to the estimate because spending varies a lot between visitors; revenue needs more data to pin down than conversion does, and that is normal. The guardrail passes: support contacts were 2.87% against 2.88%, and even the pessimistic end of the interval (an increase of 0.33 points) is below our 0.5-point margin. Had the upper limit crossed the margin, the correct statement would be "we cannot rule out harm", which is a reason to wait or to look closer.

Step 5: Segments, with care

Stakeholders will ask: "Does it work on mobile?" A segment analysis is useful for generating ideas and for planning rollouts, but it comes with two risks. Each segment has fewer visitors, so its interval is wider. And the more segments you slice, the more likely one looks special by chance. Decide the few segments that matter before looking (here, device), report all of them, and label the findings as exploratory.

Python Step 5: effect by device with intervals, and a forest plot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plot makes the pattern visible at a glance: the overall effect is +0.82 points; mobile shows +1.36 (interval 0.64 to 2.09); desktop shows +0.31 and tablet -0.75, and both of those intervals include zero. The tablet interval is very wide because there are only about 2,700 tablet visitors. But do not conclude that "it only works on mobile" from intervals alone. The right test is the interaction: does the effect on mobile differ from the effect on desktop? The last line reports it: p = 0.079. That is suggestive, but it does not clear the usual 5% bar, so the claim "mobile is different" is not established by this test. A wide interval that includes zero means "not enough data", not "no effect". Segments that were not planned remain hypotheses for the next experiment.

Step 6: The decision memo

Finish with a note a product manager can act on in two minutes. Lead with the recommendation, give the numbers with intervals, state what was checked and what could not be concluded.

Recommendation: ship the new checkout page to all visitors and monitor the support rate.

Result. Conversion rose from 7.7% (control) to 8.5% (variant) over 40,000 visitors, an increase of 0.8 percentage points (about 11% relative); the 95% interval for the difference runs from +0.3 to +1.4 points. Revenue per visitor rose by 0.42 (from 3.34 to 3.76, interval +0.12 to +0.71), driven by more buyers rather than larger orders.

Health. Assignment was balanced (no sample ratio mismatch), covariates matched, and the data passed integrity checks. The support-contact guardrail passed: the increase, if any, is below 0.33 points, inside the 0.5-point margin.

Segments (exploratory). The gain is largest on mobile (+1.4 points); desktop and tablet results are inconclusive, and the test that mobile differs from desktop is only suggestive (p = 0.08). We did not plan these cuts as decisions.

Risks and next steps. One test, one period: novelty effects may fade, so keep a 5% holdout for four weeks after launch. Run a desktop-focused follow-up if desktop revenue matters.

Check each figure in the memo against the output above before you send it. The numbers in a memo you write for real data must come straight from the analysis, never from memory.

Note: We know the simulation's truth: B is better on mobile only, by an 18% relative increase, and the same as A elsewhere. Mobile is 55% of visitors, so the overall effect is a blend of about 10% relative, and the measured 10.7% is close to it. The analysis recovered this well, and it also shows why "the average effect" can hide big differences between groups.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is the plan (metric, sample size, decision rule) written before the test starts?

    Choices made after seeing the data inflate false positives. A written plan keeps the test confirmatory.
  2. 2. The arms contain 20,450 and 19,550 visitors under a planned 50/50 split, and the sample ratio test gives p < 0.0001. What should you do?

    A sample ratio mismatch signals that randomization or data collection broke, which can bias the comparison in ways statistics cannot fix.
  3. 3. Why prefer the confidence interval for the difference over only reporting p = 0.01?

    A decision needs the size of the effect and how uncertain it is. The p-value alone does not give either.
  4. 4. Support contacts are a guardrail with a 0.5-point margin. Which result lets you say the guardrail passed?

    Not finding a significant difference is different from showing harm is small. Non-inferiority requires the pessimistic end of the interval to stay inside the margin.
  5. 5. On tablets, the interval for the lift includes zero. What can you conclude?

    A wide interval around zero reflects uncertainty. To claim that effects differ between segments you would test the interaction, ideally with more data.

Summary

  • Write the plan first: hypothesis, one primary metric, guardrails, minimum effect, error rates, sample size and the decision rule.
  • Check health before results: sample ratio, covariate balance and data integrity.
  • Report the effect as a difference with a 95% interval, plus relative lift; use bootstrap intervals for skewed metrics like revenue.
  • Test guardrails for harm against a stated margin, not for "no significant difference".
  • Treat segments as exploratory unless planned; test interactions and respect wide intervals.
  • End with a memo that leads with the recommendation, states the evidence with uncertainty, and names risks and next steps.

Next: Project: House Price Model.