Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 38 ยท Projects and Practice

Project: An A/B Test Report with Honest Uncertainty

Analyse an A/B test with seaborn: compare conversion and revenue, draw bootstrapped confidence intervals, show the distribution of differences and report a clear result.

A company shows half its visitors a new checkout page (B) and the other half the old one (A). Did the new page help? This is an A/B test, and it is the perfect place to use seaborn's statistical features: intervals, estimators and distributions. The aim is a report that shows the result and how sure we can be, rather than a single number that looks more certain than it is.

You will compare conversion rates (yes or no), compare revenue per visitor (a skewed number), use the bootstrap to get an interval for the difference, and write a conclusion in plain words. The data is synthetic with a built-in true effect, so you can check whether the analysis finds it.

The data

Python Simulate the experiment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The observed lift is two percentage points (10.0% against 12.0%). Is it real, or just luck in who happened to convert? A single difference cannot answer that; an interval can.

Conversion rate with a confidence interval

For a 0/1 column the mean is the rate. pointplot and barplot bootstrap an interval for each variant:

Python Conversion rate by variant

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two intervals overlap a little, which is typical when an effect is modest. Overlap is not a test. Looking at two separate intervals is a weak way to judge a difference, so compute an interval for the difference itself.

Bootstrapping the difference

The recipe: resample visitors in A and in B separately, compute rate_B - rate_A for each resample, and repeat 5,000 times. The spread of those differences is your uncertainty.

Python Bootstrap distribution of the lift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the whole interval lies right of zero, B is credibly better. If it straddles zero, the data are consistent with "no effect". Both outcomes are reportable; the failure is claiming a win when the interval includes zero or a loss when it merely fails to exclude it. With this seed the interval sits mostly above zero, but note how close it comes to touching zero: a modest effect needs a large sample to be pinned down.

Revenue per visitor: skewed data

Revenue per visitor is mostly zeros, with a long tail of large orders. Means are sensitive to that tail, so look at the shape before comparing:

Python Distribution of order values and mean revenue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The log scale shows the order values are roughly log-normal, and that the order-size distribution is about the same in A and B. The difference in revenue per visitor therefore comes from more people buying, not from bigger orders. That is a better story than a bare "B earns more", and it only came from drawing the distributions.

Segments: does it work on mobile and desktop?

A result that holds in every segment is more convincing than one driven by a single segment. Check with catplot:

Python
sns.catplot(data=df, x="device", y="converted", hue="variant", kind="point",
            errorbar=("ci", 95), dodge=0.3, height=4, aspect=1.2)

Be careful: splitting into many segments shrinks the sample in each, widens intervals and gives you more chances to find a flukey "significant" segment. Decide the segments before looking at results.

The write-up

A good A/B report is short and states uncertainty:

Result. B converted at 12.0% against 10.0% for A, a lift of 2.0 percentage points (95% bootstrap interval roughly 0.2 to 3.8). The interval is positive but wide: the true lift could be very small. Revenue per visitor is higher for B, driven by more purchases; order values are similar. Recommendation: B is likely better; ship it, and keep monitoring for two more weeks to narrow the range.

Include: sample sizes, the metric, the interval and what it means, the main risk, and a decision. Do not hide the interval. For formal tests, see SciPy statistics and statsmodels; for the plots behind this, see error bars and confidence intervals.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Two separate 95% intervals overlap slightly. What can you conclude?

    Overlap of two intervals is not a test; the difference has its own uncertainty.
  2. 2. In the bootstrap, what do you resample?

    You resample observations with replacement within each variant and recompute the statistic.
  3. 3. Order values are skewed. What does a log scale histogram help show?

    On a log axis a log-normal distribution is symmetric, making shape and shift easy to compare.
  4. 4. Why decide the segments before looking at results?

    Every extra split is another chance for a spurious pattern.

Summary

  • Compare means or rates with bootstrapped intervals, and compute an interval for the difference.
  • Look at distributions before comparing means of skewed data; use log scales for multiplicative data.
  • Check consistency across pre-planned segments.
  • Write the result with the effect size, the interval, the limits and a recommendation.

Next: Practice: Exercises, Cheat Sheet and Common Mistakes