Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 28 ยท Hypothesis Testing

ANOVA and Multiple Testing

Compare several groups with one-way and two-way ANOVA in Statsmodels, run Tukey HSD post-hoc tests, and control false positives with Bonferroni, Holm and FDR corrections.

When you compare three or more groups, running a t-test for every pair is a trap. With 5 groups there are 10 pairs, and at the usual 5% level, you would expect a false alarm about every other time even when nothing is going on. ANOVA (analysis of variance) asks one overall question first: do the group means differ at all? Then post-hoc tests find where, and multiple-testing corrections keep the false-positive rate honest.

One-way ANOVA through a regression

In Statsmodels, ANOVA is a view of a linear model. Fit with a formula, then call anova_lm, which splits variability into "between groups" and "within groups".

Python One-way ANOVA

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The row C(group) has an F-statistic and p-value for "all group means are equal". The Residual row is the within-group noise. A small p-value tells you something differs but not what.

Post-hoc: Tukey HSD

Tukey's Honestly Significant Difference compares every pair while controlling the overall error rate. It reports each difference, its adjusted interval and whether to reject.

Python Tukey HSD pairwise comparisons

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Usually only the pairs involving group C (the genuinely different one) are flagged.

Two-way ANOVA

With two factors, the model can include an interaction: does the effect of one factor depend on the level of the other? Write C(a) * C(b).

Python Two factors with interaction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With unbalanced data (different group sizes), the type of sums of squares matters. typ=2 is a good default when there is no interaction of interest, typ=3 tests each term given all others including interactions (use it with sum-to-zero coding: C(a, Sum)).

Correcting for many tests

If you run m independent tests at level 0.05 with all nulls true, the chance of at least one false positive is 1 - 0.95^m. For 20 tests it is 64%. Corrections fix that:

Method Controls Notes
bonferroni Family-wise error rate Simple, conservative
holm Family-wise error rate Always at least as powerful as Bonferroni; a good default
fdr_bh False discovery rate Benjamini-Hochberg; for screening many hypotheses
Python multipletests with Bonferroni, Holm and FDR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without correction, chance results sneak in; corrected tests keep only the convincing ones. Choose the correction before looking at results, and apply it to the whole family of tests you ran, including the ones you did not report.

Non-parametric alternatives

If the data are strongly skewed or tiny, rank-based tests such as Kruskal-Wallis do not assume normality. SciPy provides them (see Hypothesis Tests), and you can feed their p-values to multipletests.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why not run a t-test for every pair of 5 groups at alpha = 0.05?

    Each test has its own false-positive chance; across many tests those add up, which is why ANOVA and corrections exist.
  2. 2. What does a significant overall ANOVA F-test tell you?

    ANOVA tests the global null that all means are equal.
  3. 3. Which Statsmodels function runs Tukey HSD?

    pairwise_tukeyhsd lives in statsmodels.stats.multicomp.
  4. 4. Which method controls the false discovery rate?

    Benjamini-Hochberg (fdr_bh) controls the expected share of false discoveries among rejections.

Summary

  • ANOVA tests many group means at once; fit with a formula and call anova_lm.
  • Follow a significant result with Tukey HSD for pairwise differences.
  • Include interactions in two-way designs with C(a) * C(b).
  • Use multipletests (Holm or FDR) whenever you run many tests.

Next: Logistic and Probit Regression