Skip to content
Tutorials Center
My learning

Topics

Lesson 49 of 57 ยท Applied Statistics

Multiple Testing and Effect Sizes

Understand why many tests inflate false positives, how Bonferroni, Holm and Benjamini-Hochberg corrections work, and why effect sizes matter more than p-values.

A p-value below 0.05 means: if there were no effect, data this extreme would turn up in about 1 test in 20. That is fine for one planned test. But modern analyses run dozens or thousands of tests: every metric in a dashboard, every gene in a study, every subgroup of customers. At that scale, "1 in 20" becomes a guarantee that some noise will look like a discovery. This lesson covers how to cope, and why a p-value should never stand alone.

The multiple comparisons problem

If you run m independent tests, each at ฮฑ = 0.05, and every null hypothesis is true, the probability of at least one false positive is 1 โˆ’ (1 โˆ’ 0.05)^m. That is 23% for 5 tests, 64% for 20, and 99.4% for 100. This overall chance is the family-wise error rate (FWER).

Python False positives when everything is null

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is the engine of p-hacking: testing many outcomes, subgroups, transformations or stopping rules, and reporting only what "worked". The honest remedies are to preregister the primary analysis, label everything else as exploratory, and correct for the number of tests.

Controlling FWER: Bonferroni and Holm

Bonferroni: with m tests, call a result significant only if p < ฮฑ/m (or equivalently multiply each p-value by m). It is simple and guaranteed to control the FWER, but conservative, especially for large m.

Holm's step-down method is uniformly more powerful and just as valid: sort the p-values pโ‚โ‚โ‚Ž โ‰ค โ€ฆ โ‰ค pโ‚โ‚˜โ‚Ž; compare pโ‚แตขโ‚Ž with ฮฑ/(m โˆ’ i + 1), starting from the smallest; stop at the first one that fails. Prefer Holm over plain Bonferroni.

Controlling the false discovery rate: Benjamini-Hochberg

When you screen thousands of candidates (genes, features) it is too strict to demand zero false positives. Instead control the false discovery rate (FDR): the expected proportion of false positives among the results you declare significant. With the Benjamini-Hochberg (BH) procedure: sort the p-values, find the largest i with pโ‚แตขโ‚Ž โ‰ค (i/m)ยทq, and call the i smallest significant. If q = 0.05, about 5% of your discoveries are expected to be false.

Method Controls Strictness Typical use
Bonferroni FWER very strict few tests, costly false positives
Holm FWER strict, more powerful few to moderate tests
Benjamini-Hochberg FDR lenient many tests, exploratory screening
Python Compare corrections on 50 tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Uncorrected testing produces false positives; Bonferroni and Holm remove them but may miss real effects; BH usually finds more real effects at the cost of a small, controlled share of false ones.

Statistical vs practical significance

A p-value mixes two things: the size of the effect and the amount of data. With a huge sample a negligible difference gets a tiny p-value; with a small sample a large, important difference may not reach 0.05. So always report an effect size, a standardized measure of magnitude, plus a confidence interval.

Common effect sizes:

  • Cohen's d (difference of two means in standard deviations): d = (xฬ„โ‚ โˆ’ xฬ„โ‚‚)/s_pooled. Rough benchmarks: 0.2 small, 0.5 medium, 0.8 large.
  • Pearson r or Rยฒ for relationships; eta-squared (ฮทยฒ) for ANOVA, the share of variance explained.
  • Odds ratio or risk difference for binary outcomes.
  • Rank-biserial correlation or the probability of superiority for nonparametric tests (Nonparametric Tests).

The benchmarks are conventions. The true question is "does an effect this big matter here?" A 2% improvement in a conversion rate may be huge for a store with millions of visitors and irrelevant elsewhere.

Python Significance is not importance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first result has a tiny p-value but a trivial effect (d near 0.03). The second has a large effect (d above 1) yet a very wide interval, telling you the study was small; with other random draws it could easily miss p < 0.05.

A checklist for honest reporting

  1. State your hypotheses and the primary outcome before looking at the data.
  2. Report how many tests you ran, including ones you dropped.
  3. Correct for multiplicity when you test many things (FWER for confirmatory claims, FDR for screening).
  4. Show effect sizes with confidence intervals, not only p-values.
  5. Replicate surprising findings in new data.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You run 20 independent tests at alpha = 0.05 and all null hypotheses are true. What is the approximate chance of at least one false positive?

    1 - 0.95^20 is about 0.64.
  2. 2. What does the Benjamini-Hochberg procedure control?

    BH controls the false discovery rate, which suits large screening studies where a small share of false leads is acceptable.
  3. 3. A study with 100,000 users finds a significant difference of d = 0.02. Best description?

    A very large sample can make a negligible effect significant, so effect size and context decide importance.
  4. 4. Why is Holm preferred over plain Bonferroni?

    Holm's step-down procedure gives the same FWER guarantee with more power than Bonferroni.

Summary

  • Running many tests inflates false positives: 1 โˆ’ (1 โˆ’ ฮฑ)^m.
  • Bonferroni and Holm control the family-wise error rate; Benjamini-Hochberg controls the false discovery rate and suits large screens.
  • p-hacking comes from unreported flexibility; preregister and report all tests.
  • Always pair p-values with an effect size and a confidence interval.

Next: Project: Exploratory Data Analysis