Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 57 ยท Hypothesis Testing

ANOVA: Comparing Three or More Means

Compare several group means with one-way ANOVA: sums of squares, the F statistic, assumptions, Tukey post-hoc tests, effect size and a look at two-way ANOVA, with Python.

You have tested three teaching methods and want to know whether average scores differ. You could run a t-test on every pair (A vs B, A vs C, B vs C), but that inflates the false-positive rate: with 3 tests at 5% each the chance of at least one false alarm is about 14%, and with 10 groups (45 pairs) it is about 90%. Analysis of variance (ANOVA) compares all the means in a single test and keeps the Type I error at the chosen level.

The idea

Despite its name, ANOVA tests means by comparing two kinds of variability:

  • Between-group variability: how far the group means are from the overall mean.
  • Within-group variability: how much individual values vary around their own group mean (the noise).

If the group means are truly equal, the between-group variation should be about as big as the noise. If it is much bigger, the means differ. One-way ANOVA tests

TEXT
H0: mu1 = mu2 = ... = mu_k      H1: at least one mean differs

Sums of squares and the F statistic

Let k be the number of groups and N the total number of observations.

TEXT
SS_total   = sum over all values of (x - grand mean)^2
SS_between = sum over groups of n_j (group mean - grand mean)^2
SS_within  = sum over all values of (x - own group mean)^2
SS_total   = SS_between + SS_within

Dividing each by its degrees of freedom gives mean squares:

TEXT
MS_between = SS_between / (k - 1)
MS_within  = SS_within / (N - k)
F = MS_between / MS_within

Under H0, F follows the F distribution with (k - 1, N - k) degrees of freedom. Large F means the group means are spread out compared with the noise; the test is right-tailed.

Python One-way ANOVA by hand and with SciPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The summary is often laid out as an ANOVA table with columns Source, SS, df, MS, F and p.

Effect size

Eta squared is the proportion of the total variation explained by the group:

TEXT
eta^2 = SS_between / SS_total

Rules of thumb: about 0.01 small, 0.06 medium, 0.14 large.

Post-hoc tests: which groups differ?

A significant ANOVA only says that at least one mean differs. To find out which pairs, run a post-hoc procedure that controls the overall error rate, such as Tukey's HSD.

Python Tukey's HSD after a significant ANOVA

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The printed table gives, for each pair, the mean difference, a p-value adjusted for multiple comparisons, and a confidence interval.

Assumptions

  1. Independent observations (randomly assigned or sampled).
  2. Approximately normal residuals within each group (ANOVA is fairly robust with equal group sizes and moderate n).
  3. Equal variances across groups (homogeneity). Check with Levene's test; if violated, use Welch's ANOVA, or the nonparametric Kruskal-Wallis test.
Python Checking assumptions and the nonparametric alternative

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why many t-tests are not the answer

Python The multiple-comparisons trap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With no real differences at all, pairwise testing "finds" one in more than a third of the experiments, while ANOVA stays near 5%.

Two-way ANOVA in brief

Two-way ANOVA studies two factors at once, for example teaching method and class size. It reports a main effect of each factor and an interaction effect: whether the effect of one factor depends on the level of the other. Total variation splits further:

TEXT
SS_total = SS_A + SS_B + SS_(A x B) + SS_error

Each effect gets its own F test. A significant interaction means you should interpret main effects with care, since "the best method" depends on class size. In Python this is done with statsmodels (ols plus anova_lm); SciPy has only the one-way version.

flowchart TD
    A["Compare means across groups"] --> B{"How many factors?"}
    B -- "One factor, 3+ levels" --> C["One-way ANOVA"]
    C --> D{"Significant?"}
    D -- "Yes" --> E["Post-hoc (Tukey) to find which pairs"]
    B -- "Two factors" --> F["Two-way ANOVA with interaction"]

From a single F test to the follow-up analyses.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why not just run all pairwise t-tests for 5 groups?

    Ten pairwise tests at 5% each have a much higher family-wise error rate than 5%.
  2. 2. With k = 4 groups and N = 40 observations in total, the degrees of freedom of F are:

    df between = k - 1 = 3; df within = N - k = 36.
  3. 3. A significant one-way ANOVA tells you:

    It rejects the claim that all means are equal; a post-hoc test finds where.
  4. 4. If SS_between = 30 and SS_total = 120, eta squared is:

    eta^2 = 30 / 120 = 0.25: 25% of the variation is explained by group.

Summary

  • ANOVA tests whether several means are equal using F = MS_between / MS_within.
  • It avoids the inflated error rate of many pairwise t-tests.
  • After a significant result, use Tukey's HSD (or similar) to see which groups differ; report eta squared.
  • Check independence, normality and equal variances; use Welch's ANOVA or Kruskal-Wallis when they fail.
  • Two-way ANOVA adds a second factor and an interaction term.

Next: back to the Probability and Statistics course.