Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 37 ยท Sequential and Bayesian Tests

Bayesian A/B Testing with the Beta-Binomial Model

Run a Bayesian A/B test on conversion rates: Beta priors and posteriors, probability that B beats A, credible intervals and expected loss, with runnable Python examples.

The frequentist test asks: "if there were no difference, how surprising would this data be?" The Bayesian test asks the question stakeholders actually have: "given this data, how likely is it that B is better than A, and how much could I lose if I am wrong?"

For conversion rates (a user either converts or does not), one model covers almost everything: the Beta-Binomial model. It is short enough to understand completely and fast enough to run inside a web page.

Prior, likelihood, posterior

Before seeing data you describe your belief about a conversion rate p with a Beta(a, b) distribution. Beta(1, 1) is flat: every rate from 0 to 1 is equally plausible. Beta(2, 38) says "probably around 5%, with real uncertainty".

If x of n users convert, the posterior is another Beta:

TEXT
posterior = Beta(a + x, b + (n - x))

That is the whole update: add conversions to a and non-conversions to b. Because the prior and posterior are the same family, the Beta is called conjugate to the binomial.

Bayesian update in three steps

Prior Beta(a, b): what we believed
Data x conversions out of n
Posterior Beta(a + x, b + n - x)

More data, narrower posterior

1 / 2
Python Posteriors for two variants

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The interval printed here is a credible interval. It means what people usually think a confidence interval means: given the model and prior, there is a 95% probability the true rate lies inside.

Probability that B beats A

Two Beta curves overlap, so we cannot compare them with a single subtraction. The easy route is Monte Carlo: draw many samples from each posterior and count how often B's draw is larger.

Python Chance that B beats A, uplift and expected loss

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Expected loss is the most useful number on the page. It weighs how likely you are to be wrong by how much you lose when wrong. A common decision rule: ship B when its expected loss drops below a threshold you can live with, for example 0.1 percentage points of conversion.

The effect of the prior

With thousands of users the prior hardly matters. With tiny samples it matters a lot, which is a feature when you have real knowledge and a danger when you invent it.

Python Flat prior versus informed prior

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A stronger prior pulls both posteriors toward 8%, so the observed gap counts for less. Choose priors from historical data (past tests on this page, or the portfolio average) and write them down before the test. If two reasonable priors lead to opposite decisions, the data is not yet enough.

A Bayesian stopping rule is not immune to peeking. If you check "P(B > A) > 95%" every day and stop at the first hit, your long-run false-ship rate is higher than you think. Pair the rule with a minimum sample size, a loss threshold, and a planned maximum duration. See The Peeking Problem.

Beyond conversion rates

The same recipe works whenever a conjugate pair exists: Gamma-Poisson for counts per user, Normal-Normal for averages with known variance. For revenue and other skewed metrics people usually fall back on bootstrapping or a Normal approximation of the mean, which works well once samples are large.

Frequentist versus Bayesian, honestly

  • With a flat prior and a large sample the numbers agree closely: the Bayesian P(B > A) is about one minus the one-sided p-value.
  • Bayesian outputs are easier to explain and to combine with costs.
  • Frequentist designs give clean error-rate guarantees under a fixed plan. Bayesian methods give you posterior beliefs; guaranteeing long-run error rates needs extra simulation work.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In the Beta-Binomial model, 30 of 400 users convert and the prior is Beta(1, 1). What is the posterior?

    Add conversions (30) to a and non-conversions (370) to b: Beta(1 + 30, 1 + 370) = Beta(31, 371).
  2. 2. What does expected loss measure?

    It combines the probability of being wrong with the size of the mistake.
  3. 3. Which situation makes the choice of prior matter most?

    With little data the prior carries a large share of the information in the posterior.
  4. 4. Is it safe to stop a Bayesian test the first time P(B > A) passes 95%, checked daily?

    Giving noise many chances to cross a threshold inflates mistakes regardless of the statistical philosophy.

Summary

  • The Beta-Binomial model updates a conversion-rate belief by adding conversions to a and misses to b.
  • Monte Carlo draws from the two posteriors give P(B > A), uplift intervals and expected loss.
  • Expected loss turns the decision into "is the risk small enough to ship?".
  • Priors matter with small samples; set them from history and before the test.
  • Bayesian stopping rules still need a minimum sample size and a loss threshold.

Next: Multi-Armed Bandits and Thompson Sampling