Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 36 ยท Projects and Practice

Project: A Bayesian A/B Test

Analyze a conversion experiment end to end: Beta posteriors, probability that B beats A, expected loss, a region of practical equivalence, and the danger of peeking.

A product team tests a new checkout button (variant B) against the current one (variant A). Visitors are split at random; each either buys or does not. The team wants to know: should we ship B? In this project you build the full Bayesian answer: posteriors for both conversion rates, the probability that B is better, how much is at stake if you are wrong, and a decision rule that behaves when the data arrive over time.

Everything uses the Beta-binomial model from the conjugate models chapter: with a Beta(a, b) prior and k conversions out of n visitors, the posterior is Beta(a + k, b + n - k).

Step 1: posteriors and the probability B beats A

Data: A had 4,000 visitors and 480 purchases (12.0%). B had 4,000 visitors and 520 purchases (13.0%). The prior is Beta(2, 18) for each: a weakly informative belief that conversion is near 10%.

Python Posterior of each rate and of the lift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The probability that B beats A is around 0.9: favorable, but not overwhelming, because a one-point lift on 4,000 visitors per arm is within the range of noise. A p-value answers a different question ("how surprising is this if there were no difference?"); P(B > A) is the direct statement most decision makers want.

Step 2: expected loss

Whether to ship depends on how much you lose if you are wrong, not only on the probability of being wrong. The expected loss of choosing B is the average amount by which A would have been better, over the posterior:

TEXT
expected loss of choosing B = E[ max(pA - pB, 0) ]
expected loss of choosing A = E[ max(pB - pA, 0) ]
Python Expected loss and a ROPE

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A ROPE (region of practical equivalence) states in advance which differences are too small to care about. Instead of testing "is the difference exactly zero?", you ask how much posterior mass lies inside, above and below the region. A decision rule might be: ship B if its expected loss is below a threshold that the business can tolerate. The thresholds come from costs and benefits, not statistics, so agree on them before the experiment.

Step 3: the danger of peeking

Many teams check results daily and stop when P(B > A) crosses 95%. With frequentist p-values this inflates false positives dramatically. How do Bayesian posteriors behave? We simulate experiments where A and B are identical (both 10%), look after every 200 visitors per arm up to 4,000, and count how often the stop rule (P(B > A) above 0.95 or below 0.05) fires at some point.

Python Peeking when there is no real difference

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stopping the first time the posterior crosses a threshold in either direction gives a much higher false-winner rate than looking once, because you get many chances for noise to cross the line. This is a property of the decision rule (which depends on the data you saw and when), not of Bayes' theorem. Practical fixes: use stricter thresholds when looking often, set a minimum sample size, decide via expected loss with a ROPE, or calibrate the stopping rule by simulation exactly as we just did.

Reporting the result

A clear write-up includes:

  • The prior and why it was chosen, with a sensitivity check using a flat Beta(1, 1) and a skeptical one.
  • Posterior means and 95% intervals for both rates and for the lift.
  • P(B > A), the expected loss for each choice and the ROPE probabilities.
  • The stopping rule that was fixed in advance, and the sample size reached.
  • What would change the decision (for example, the cost of the new button).

Extensions

  • Add a second metric such as revenue per visitor, which is skewed; model it with a log-normal or gamma hurdle.
  • Run the analysis per segment with a hierarchical model so small segments are shrunk toward the overall effect.
  • Compare more than two variants and compute P(each variant is best) by counting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What posterior do you get for a conversion rate with a Beta(a, b) prior and k conversions in n trials?

    The Beta prior is conjugate to the binomial likelihood: add successes to a and failures to b.
  2. 2. What is the expected loss of shipping B?

    It weighs how likely and how large the mistake would be.
  3. 3. What does a ROPE specify?

    Differences inside the ROPE are too small to matter for the decision.
  4. 4. Why does peeking inflate false winners even with posterior probabilities?

    The stopping rule's error rate depends on how often you look; simulate it and calibrate the threshold.

Summary

  • Beta posteriors give each rate and the lift, with P(B > A) computed by counting draws.
  • Expected loss and a ROPE connect the posterior to the decision, with thresholds set beforehand.
  • Peeking raises the rate of false winners, so calibrate stopping rules by simulation.
  • Report priors, sensitivity, intervals, the stopping rule and the decision logic.

Next: Project: Hierarchical Ratings