Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 37 ยท Hypothesis Tests and Metrics

Multiple Comparisons and False Discoveries

Learn why testing many metrics, segments or variants inflates false positives, and fix it with Bonferroni, Holm and Benjamini-Hochberg corrections, shown in Python.

Suppose you test one metric at alpha = 0.05. There is a 5% chance of a false positive. Now suppose your dashboard shows 20 metrics, or you slice by 10 countries and 5 devices. Even if the change does nothing at all, you should expect about one "significant" result by pure luck. This is the multiple comparisons problem, and it is the reason so many "surprising" experiment findings do not replicate.

How fast the risk grows

If you run m independent tests, each at level alpha, the chance of at least one false positive is 1 - (1 - alpha)^m. This is the family-wise error rate (FWER).

Python False positives grow with the number of tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With 20 metrics the chance of at least one false "win" is about 64%. A p-value of 0.04 on one of twenty metrics is not much evidence at all.

Where the problem hides

  • Many metrics: checking every number on the dashboard.
  • Many segments: "it worked for iOS users in Germany!"
  • Many variants: an A/B/C/D/E test compared pairwise.
  • Many looks: checking the result every day (the peeking problem).
  • Many experiments: across a company, some will be false wins; this is why replication matters.

The cleanest defense is design: one primary metric, a short list of pre-registered secondary metrics, and segments only as exploration. When you do need many comparisons, apply a correction.

Controlling the family-wise error rate

Bonferroni: use alpha / m for each test. Simple and safe, but conservative (you lose power as m grows).

Holm (step-down): sort p-values from smallest to largest, compare the smallest to alpha/m, the next to alpha/(m-1), and so on, stopping at the first failure. It is uniformly more powerful than Bonferroni and just as valid, so prefer it.

Controlling the false discovery rate

When you test many hypotheses and can tolerate a few false leads (for example screening many segments), control the false discovery rate (FDR): the expected share of your discoveries that are false. The Benjamini-Hochberg (BH) procedure: sort p-values, find the largest k with p_(k) <= (k/m) * q, and reject the first k. It is more powerful than FWER methods.

Python Bonferroni, Holm and Benjamini-Hochberg

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Five metrics look significant on raw p-values. After correction, only the strongest survive; BH keeps the most and Bonferroni the fewest. That ordering (BH most liberal, Bonferroni strictest) is typical.

Simulation: correction restores the error rate

Python Does Holm control the error rate?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Raw testing yields a false positive in about two thirds of experiments; Holm brings it back near 5%.

A/B/n tests

With one control and k treatments you make k comparisons against control. Either adjust (Holm or Dunnett's test, which is designed for "each versus control") or run an omnibus test first (Chi-Square) and follow up only if it is significant. State the plan beforehand.

Practical guidance

flowchart TD
    A["Many comparisons?"] --> B{"Confirmatory<br/>decision?"}
    B -- yes --> C["Pre-register few hypotheses<br/>FWER control (Holm)"]
    B -- no --> D["Exploration: FDR (BH)<br/>treat hits as leads"]
    C --> E["Replicate surprising wins"]
    D --> E

Confirm decisions with strict control; treat exploratory hits as hypotheses for the next experiment.

  • Declare one primary metric: it needs no correction, because it is a single test.
  • Guardrails are different: you want to detect harm, so a missed harm is worse than a false alarm. Many teams use a looser threshold or no correction for guardrails.
  • Beware "segment hunting." If a result appears only in one slice and was not predicted, call it a lead and rerun it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. If you test 20 true-null metrics at alpha = 0.05, what is the approximate chance of at least one false positive?

    1 - 0.95^20 is about 0.64. The more tests, the likelier at least one is significant by chance.
  2. 2. Which procedure is uniformly more powerful than Bonferroni while still controlling the family-wise error rate?

    Holm compares sorted p-values to progressively looser thresholds, rejecting at least everything Bonferroni does.
  3. 3. What does the Benjamini-Hochberg procedure control?

    BH controls the false discovery rate, allowing more discoveries when you can tolerate a small share of false ones.
  4. 4. Which design choice most reduces the need for correction?

    A single pre-registered primary metric is one test, so there is no multiplicity for the decision.

Summary

  • Testing many metrics, segments, variants or looks inflates false positives: 20 tests give about a 64% chance of at least one.
  • Prefer one primary metric; for the rest, use Holm (FWER) or Benjamini-Hochberg (FDR).
  • Treat unplanned segment findings as leads, not conclusions, and replicate them.
  • Guardrails are about detecting harm, so correct them gently.

Next: Statistical Power and Error Types