Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 38 ยท Statistics

Hypothesis Testing

Run t-tests, Mann-Whitney, chi-square, ANOVA and normality tests with scipy.stats, read p-values and effect sizes, correct for multiple tests and use permutation_test.

You measure two groups and their averages differ. Is the difference a real effect, or just the luck of which people landed in which group? A hypothesis test answers one narrow question: if there were no real effect at all, how often would chance alone produce a difference at least this large? That probability is the p-value. A small p-value says the data would be surprising under "no effect", so you reject that assumption.

scipy.stats implements the classic tests behind a consistent interface: pass in the data, get back a result object with a statistic and a pvalue. This lesson shows the tests you will use most often, how to pick between them, why a p-value is never the whole story, and how to run a test with no distribution assumptions at all using resampling. It builds on Distributions and Descriptive Statistics.

flowchart TD
    Q["What are you comparing?"] --> A["Two groups of numbers"]
    Q --> B["Three or more groups"]
    Q --> C["Counts in categories"]
    A -->|"roughly normal"| T["ttest_ind (Welch)"]
    A -->|"skewed or ordinal"| M["mannwhitneyu"]
    A -->|"same people twice"| P["ttest_rel / wilcoxon"]
    B --> F["f_oneway, then tukey_hsd"]
    C --> X["chi2_contingency"]

A rough map from the shape of the question to the test. Each branch has a more robust or more specialized relative.

The recipe and the t-test

Every test follows the same steps: state a null hypothesis (no difference), choose a significance level alpha in advance (0.05 is the habit, not a law), compute the test statistic, and compare the p-value with alpha.

The workhorse for two groups of numbers is the t-test. Use ttest_ind for independent groups. Pass equal_var=False to get Welch's test, which does not assume equal variances and is a safe default. Use ttest_rel when the two measurements belong to the same subjects (before and after), and ttest_1samp to compare one group with a fixed value.

Python Comparing two groups with a t-test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The alternative argument ("two-sided", "less", "greater") fixes the direction of your claim before you look at the data. Choosing "greater" after seeing which mean is bigger quietly halves your p-value and is a form of cheating. The result object also carries a confidence_interval() for the difference, which is usually more informative than the p-value alone.

When the data is not normal: Mann-Whitney

The t-test is fairly forgiving with moderate or large samples, but it can mislead on small, heavily skewed samples or on ordinal data such as star ratings. The Mann-Whitney U test compares the ranks of the values instead of the values themselves. It asks whether a random observation from one group tends to be larger than one from the other, so a few extreme values cannot dominate it.

Python Skewed data and Mann-Whitney

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For paired data the rank-based counterpart is stats.wilcoxon(after, before). As a rule: reach for a rank test when the sample is small and clearly non-normal, or when the numbers are ranks to begin with.

Counts: the chi-square test

When the data is counts in categories you cannot average anything, so you compare observed counts with the counts you would expect if the null hypothesis held. chisquare(observed, expected) tests one set of counts against a fixed distribution (is this die fair?). chi2_contingency(table) tests whether two categorical variables are independent, for example whether conversion depends on the page variant. It returns the statistic, p-value, degrees of freedom and the expected table.

Python Is the die fair? Does the variant matter?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The test is a large-sample approximation. A common guideline is that every expected count should be at least about 5; for small 2x2 tables use stats.fisher_exact(table) instead.

Three or more groups: ANOVA

Running a t-test between every pair of three groups inflates your error rate (more on that below). One-way ANOVA tests the single question "are all the group means equal?" in one go, using the ratio of between-group to within-group variability (the F statistic). A small p-value says at least one group differs, not which. For that, follow up with a post-hoc test such as tukey_hsd, which compares every pair while controlling the overall error rate.

Python ANOVA and Tukey's post-hoc test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Checking assumptions: normality tests

Many tests assume roughly normal data. stats.shapiro(x) (best for small to medium samples) and stats.normaltest(x) (based on skewness and kurtosis) test the null hypothesis "this sample came from a normal distribution". A small p-value suggests it did not.

Use them with care. With a huge sample, even a trivially small departure from normality gives a tiny p-value; with a tiny sample, a clearly skewed variable can pass. A Q-Q plot (stats.probplot) shows how the data departs from normal and is usually the better guide.

Python Shapiro-Wilk on normal and skewed data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What a p-value is not

A p-value is not the probability that the null hypothesis is true, not the probability that you made a mistake, and not the size of the effect. A tiny effect measured on a huge sample gives a tiny p-value; a large effect measured on a few people may not reach 0.05. Always report an effect size and an interval next to the p-value. A common effect size for two means is Cohen's d: the difference in means divided by the pooled standard deviation. Rough guidance: 0.2 is small, 0.5 medium, 0.8 large.

Python Significant is not the same as important

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The huge sample produces an astronomically small p-value for a difference of 0.02 standard deviations, while the small sample has a large effect and a much weaker p-value. Judge the size of the effect first and the p-value second.

Many tests at once: multiple comparisons

If you run 20 tests on pure noise with alpha = 0.05, you expect about one false "discovery". The more questions you ask, the more likely one of them looks significant by luck. Two standard fixes:

  • Bonferroni: compare each p-value with alpha / m for m tests (strict, simple, controls the chance of any false positive).
  • Benjamini-Hochberg (stats.false_discovery_control): controls the share of false discoveries among the ones you report, with more power when you test many hypotheses.
Python Two hundred tests on pure noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Permutation tests: no formulas needed

If the null hypothesis says the group labels do not matter, then shuffling the labels should produce differences as large as the observed one reasonably often. A permutation test does exactly that: compute your statistic on thousands of random relabelings and see where the real value falls. It works for any statistic (median, ratio, trimmed mean) and needs no normality assumption. stats.permutation_test takes your data as a tuple of samples and a function of those samples.

Python A permutation test on the difference of medians

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With few observations permutation_test tries every relabeling (an exact test). With many, it samples n_resamples of them, so fix random_state for repeatable p-values.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a p-value of 0.03 tell you?

    A p-value is computed assuming the null hypothesis is true. It is not the probability of the hypothesis and says nothing directly about effect size.
  2. 2. Which call gives Welch's t-test, which does not assume equal variances?

    ttest_ind assumes equal variances by default. equal_var=False switches to Welch's version. ttest_rel is for paired data.
  3. 3. You compare three groups with ANOVA and get p = 0.001. What can you conclude?

    ANOVA tests the single null hypothesis that all means are equal. Follow up with tukey_hsd to find which pairs differ.
  4. 4. Why correct for multiple comparisons?

    With 20 independent tests at alpha 0.05 you expect about one false positive. Bonferroni and Benjamini-Hochberg adjust for this.
  5. 5. A study with 200,000 people finds p < 0.001 and Cohen's d = 0.02. How should you describe it?

    A large sample detects tiny effects. The effect size shows that the difference is very small even though the p-value is.

Summary

  • A hypothesis test asks how surprising the data is if there were no effect; the p-value measures that surprise and is not the probability the null is true.
  • ttest_ind(..., equal_var=False), ttest_rel and ttest_1samp compare means; mannwhitneyu and wilcoxon are rank-based alternatives for skewed or ordinal data.
  • chisquare and chi2_contingency test counts; fisher_exact handles small 2x2 tables.
  • f_oneway tests three or more means, followed by tukey_hsd; kruskal is the rank-based version.
  • shapiro and normaltest check normality, but plots such as Q-Q plots are more informative.
  • Report an effect size (such as Cohen's d) and a confidence interval, not just p.
  • Correct for many tests with Bonferroni or false_discovery_control; use permutation_test for assumption-free inference on any statistic.

Next up: relationships between variables and uncertainty by resampling in Correlation, Regression and Bootstrap.