Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Pitfalls and Ethics

Multiple Comparisons and Segment Hunting

Why testing many metrics and segments produces false wins, and how Bonferroni, Holm and Benjamini-Hochberg corrections control the damage, with runnable Python simulations.

A test with one metric and a 5% threshold gives you a 5% chance of a false alarm when nothing changed. A test with 20 metrics looked at in the dashboard gives you about a 64% chance that at least one of them lights up, even if the change did nothing. Add five segments (country, device, new vs returning...) to each metric and a "significant" result is nearly guaranteed.

This is the multiple comparisons problem. The rule behind it is simple: the more things you look at, the more likely something looks special by chance.

How fast the risk grows

If tests are independent and each has a 5% false positive rate, the probability of at least one false positive among m tests is 1 - 0.95^m.

Python Chance of at least one false positive

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Where multiplicity hides

Many metrics Clicks, revenue, retention...
Many segments Country, device, plan
Many variants A/B/C/D/E
Many looks See the peeking lesson

Each one multiplies the chances of a false win

1 / 2

Corrections

Two different error rates can be controlled:

  • Family-wise error rate (FWER): the probability of any false positive. Strict. Use when one false claim is costly (a shipped decision depends on every result).
  • False discovery rate (FDR): the expected fraction of your discoveries that are false. Looser, more powerful. Use when you screen many metrics or segments and will follow up on the hits.

Three standard methods, with m tests and sorted p-values p(1) <= ... <= p(m):

  • Bonferroni: call a result significant if p < alpha / m. Simple, controls FWER, conservative.
  • Holm: step-down version of Bonferroni, always at least as powerful, same FWER guarantee. Compare p(i) with alpha / (m - i + 1) and stop at the first failure.
  • Benjamini-Hochberg (BH): find the largest i with p(i) <= (i / m) * alpha and reject all of p(1)..p(i). Controls FDR.
Python Compare the corrections on one experiment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without correction five metrics are "significant"; Bonferroni and Holm keep only the strongest, and BH (which accepts a small share of false discoveries) keeps four. Which one you want depends on the cost of a wrong claim.

Do the corrections do what they promise?

Python False discoveries with and without correction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When every null is true, both Bonferroni and BH hold the chance of a false positive near or below 5%, while the uncorrected analysis hits about 64%.

A practical policy

  1. Name one primary metric (the overall evaluation criterion) before the test. The ship decision rests on it with the ordinary 5% threshold. See choosing metrics.
  2. Treat secondary metrics as supporting evidence, corrected with BH if you screen many of them.
  3. Guardrail metrics are checked for harm; use a stricter threshold or a non-inferiority margin, and decide in advance how many you track.
  4. Segment analysis is exploratory. Only pre-registered segments (for example, "mobile users") are confirmatory. A surprise segment is a hypothesis for the next test, not a result.
  5. Multiple variants: compare each to control with Dunnett's test or Holm, not a pile of uncorrected t-tests.
  6. Report how many things you looked at, not only what was significant.

"The change hurt desktop users in Germany on Tuesdays" is the kind of sentence segment hunting produces. If you slice finely enough, noise will always give you a story. Make the story testable by running a follow-up experiment.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You track 20 independent metrics at a 5% level and nothing really changed. About how likely is at least one false positive?

    1 - 0.95^20 is about 0.64.
  2. 2. What does the false discovery rate (FDR) control?

    FDR limits the proportion of false findings among the results you declare significant.
  3. 3. Which method is always at least as powerful as Bonferroni while controlling the same error rate?

    Holm's step-down procedure controls FWER like Bonferroni but rejects at least as many hypotheses.
  4. 4. What status does a surprising result in an unplanned segment have?

    Post-hoc segment results are exploratory and should be confirmed before acting.

Summary

  • Each extra metric, segment, variant or look raises the chance of a false win; 20 tests give about a 64% chance of at least one.
  • Bonferroni and Holm control the family-wise error rate; Benjamini-Hochberg controls the false discovery rate.
  • Pick one primary metric in advance and treat everything else as supporting or exploratory.
  • Confirm surprising segment effects with a new experiment.

Next: Winner's Curse and Simpson's Paradox