Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Projects and Practice

Project: Analyze a Checkout Experiment End to End

Work through a complete A/B test analysis in Python: data checks, SRM, conversion and revenue tests, CUPED, segments with correction, Bayesian view, guardrails and a written decision.

In this project you play the analyst. Your team tested a redesigned checkout page. Users were randomly assigned to control (old page) or treatment (new page). The plan, written before launch, was:

  • Primary metric: purchase conversion (did the user buy?).
  • Secondary metric: revenue per user, capped at the 99.5th percentile.
  • Guardrail: payment error rate must not increase by more than 0.5 percentage points.
  • Pre-planned segments: device (mobile, desktop). Anything else is exploratory.
  • Decision rule: ship if conversion improves significantly at 5%, no guardrail is breached, and the revenue estimate does not point downward.

Each step below generates the same dataset (same seed), so every block runs on its own. In a real project you would load it from your warehouse once.

flowchart LR
    A["1. Data and SRM checks"] --> B["2. Primary metric"]
    B --> C["3. Revenue and CUPED"]
    C --> D["4. Segments and guardrail"]
    D --> E["5. Decision memo"]

Always check data quality first, then analyze the pre-registered metrics in order.

Step 1: Data quality and sample ratio

Before reading any effect, verify that the split is the designed 50/50 and that the data has no duplicates.

Python Step 1: SRM and data checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The split is healthy, there are no duplicates and the pre-experiment spend is balanced between arms, so we can analyze effects.

Step 2: The primary metric

Python Step 2: Conversion test with a confidence interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Report the size and interval, not just the p-value: "conversion rose by about 1.3 points (roughly 16% relative), and the interval excludes zero".

Step 3: Revenue with capping and CUPED

Python Step 3: Capped revenue and CUPED adjustment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Revenue per user is noisier than conversion, and the lift is positive, so the "revenue must not point downward" rule holds. Notice that CUPED barely changed the standard error: the correlation between pre-period spend and this period's revenue is only about 0.04, because most users do not buy at all. That is the lesson's warning in action: check the correlation first, and use a stronger covariate (for example, pre-period sessions or orders) if you want a real gain.

Step 4: Segments, guardrail and a Bayesian view

Python Step 4: Segments, guardrail and P(treatment better)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check what the numbers say about each rule in the plan. If the guardrail upper bound were above the margin, you could not simply ship: the right next step would be to investigate the payment errors.

Step 5: Write the decision memo

A good readout is one page, and it states the decision first.

  • Decision: ship the new checkout to 100%, with a 5% holdback for four weeks.
  • Evidence: conversion up by about 1.3 points (CI excludes 0), revenue per user positive, lift positive on both devices (clear on mobile, wide interval on desktop), payment errors within the margin, Bayesian probability of benefit near 1 with negligible expected loss.
  • Caveats: the test ran two weeks, so the long-term effect is unknown (check the daily lift for novelty); winners tend to be overestimated slightly (winner's curse); revenue is capped.
  • Follow-ups: monitor the payment error rate after launch; test an express-payment option next.

Try changing the numbers: set the treatment effect to 0, rerun the steps, and see how the memo changes. Also try an uneven split to trigger the SRM alert.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the project check SRM before looking at the effect?

    If the observed split deviates from the design, assignment or logging is broken and the comparison is biased.
  2. 2. In Step 3, why is the revenue cap computed on pooled data?

    Different caps per arm would change the metric differently in each arm.
  3. 3. The guardrail's upper confidence bound is above the allowed margin. What should happen?

    A guardrail exists to prevent harm; failing it blocks the launch until understood.
  4. 4. Which segments count as confirmatory in this plan?

    Only pre-planned segments are confirmatory; others are exploratory.

Summary

  • Start with data quality and SRM, then analyze the pre-registered primary metric with an interval.
  • Cap revenue with a pooled, pre-decided threshold and use CUPED to cut variance.
  • Treat pre-planned segments as confirmatory and everything else as exploratory.
  • Check guardrails against their margins and complement the frequentist result with a Bayesian view.
  • End with a one-page memo that states the decision, evidence, caveats and follow-ups.

Next: Project: Build a Mini Experimentation Engine