Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 37 ยท Sequential and Bayesian Tests

The Peeking Problem

See why checking an A/B test every day and stopping at the first significant result inflates false positives, and measure the damage with a runnable simulation.

Every product team does it: launch a test, open the dashboard the next morning, and look at the p-value. If it is below 0.05, someone says "it is significant, ship it". This habit is called peeking, and it quietly breaks the guarantee that a fixed-horizon test gives you.

A classic z-test or t-test promises: if there is no real effect, you will be fooled in at most 5% of experiments. That promise holds only if you look once, at a sample size chosen in advance. This lesson shows what happens when you look many times, and why the fix (covered in the next lessons) is a different kind of test, not more discipline.

Why one look is safe and many looks are not

Imagine the running difference between two groups as a random walk. When there is no true effect, the walk wanders around zero. At any single moment, the chance it sits beyond the 5% threshold is 5%. But the walk gets many chances to cross the line at some point. The probability that it ever crosses in 20 looks is far higher than 5%.

Peeking in one picture

No real effect A and B are identical
Look every day Each look is a new chance
Stop at first p < 0.05 Noise eventually crosses

More looks, more false alarms

1 / 2

Simulate an A/A experiment with peeking

An A/A test gives both groups the same experience, so any "win" is a false positive. We simulate 2,000 A/A experiments with 20 daily looks each and compare two policies: look once at the end, or stop at the first look where p < 0.05.

Python False positives when you peek

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single-look rate lands near 5%, as promised. The peeking rate is several times higher. Nothing about the data changed: only the decision rule did.

How bad does it get?

The damage grows with the number of looks, quickly at first and then more slowly. Here is the same idea, vectorized so it runs fast, for different numbers of looks.

Python Error rate versus number of looks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 50 looks, roughly one in three "winning" A/A tests is noise.

Peeking is only a problem when you act on what you see. Watching a dashboard to confirm that logging works or that traffic is flowing is fine. Stopping the test, shipping, or changing the plan because the p-value crossed 0.05 is the problem.

Peeking also biases the estimate

Stopping early on a lucky streak does not only create false positives. It also overstates the effect size in tests that really do have a small lift: you stop when the observed lift is high by chance. This is the winner's curse, and it explains why shipped wins often shrink after launch (see Pitfalls and Ethics).

Three honest ways to handle early looks

  1. Fix the horizon. Compute the sample size in advance, run the whole time, look once. This is simple and valid, but you cannot stop early for big wins or clear disasters.
  2. Use a sequential test. Spend your 5% error budget across planned looks (alpha spending) or use an always-valid method. You may stop early and keep the error guarantee.
  3. Use a Bayesian decision rule with a loss threshold. This changes the question but does not make peeking free; see Bayesian A/B testing.

Switching to Bayesian statistics does not remove the need to think about stopping rules. A rule such as "stop when P(B > A) exceeds 95%" also has an inflated false positive rate when you check it many times.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You run an A/A test and check the p-value daily for 20 days, stopping at the first p < 0.05. What happens to the false positive rate?

    Each look is another chance for noise to cross the threshold, so the overall error rate is higher than 5%.
  2. 2. Which behavior is actually harmful?

    Acting on repeated significance checks breaks the error guarantee. Monitoring health metrics does not.
  3. 3. Does using a Bayesian posterior probability remove the peeking problem?

    Any decision rule that gives noise many chances to look good will produce more false alarms than a single planned look.
  4. 4. Early stopping on a lucky streak tends to make the observed lift...

    You stop when the estimate is high by chance, so stopped tests overstate the true effect (winner's curse).

Summary

  • A fixed-horizon test is valid only if you analyze once at a pre-planned sample size.
  • Repeated significance checks with early stopping inflate false positives, often to 20-30% with many looks.
  • Peeking also exaggerates the effect size of stopped winners.
  • Fixes: fix the horizon, use a sequential test, or use a loss-based Bayesian rule with care.
  • Looking at dashboards is fine; making decisions from repeated looks is not.

Next: Group Sequential Tests and Alpha Spending