Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 76 ยท Statistics for Data Science

Hypothesis Testing Done Right

Understand null hypotheses, p-values, t-tests, chi-square, nonparametric tests, effect sizes and multiple comparisons, with live Python examples and common traps.

A hypothesis test answers one narrow question: "If nothing were going on, how surprising would my data be?" It is a tool for separating a real pattern from the luck of the draw. Used well, it stops you from announcing that a new page design "works" because one lucky week looked good. Used badly, it produces a stream of false discoveries that look scientific.

This lesson explains the logic in plain steps, runs the tests you will use most (t-tests, chi-square, and a nonparametric alternative), and spends real time on the traps: reading p-values wrongly, ignoring effect size, and testing many things until something "works". The function-by-function reference for SciPy is in hypothesis tests in SciPy; here the focus is on judgement. Intervals from the previous lesson will come in handy.

The logic in five steps

  1. State the null hypothesis (H0). It is the boring explanation: no difference, no relationship, no effect. Example: the new page has the same mean time on page as the old one.
  2. State the alternative (H1). The claim you would like to support: the means differ (two-sided) or the new one is higher (one-sided). Decide this before looking at the data.
  3. Choose a test statistic that measures how far the data are from H0, such as a t value.
  4. Compute the p-value: the probability of seeing a result at least this extreme if H0 were true.
  5. Decide. If p is below your chosen significance level alpha (often 0.05), you reject H0. Otherwise you fail to reject it, which is not the same as proving it.
flowchart TD
    A["Assume H0 is true<br/>(no effect)"] --> B["Collect data and<br/>compute a statistic"]
    B --> C{"How surprising is<br/>this under H0?"}
    C -->|"p small (< alpha)"| D["Reject H0:<br/>evidence of an effect"]
    C -->|"p not small"| E["Fail to reject H0:<br/>not enough evidence"]

A test measures surprise under H0; a small p-value is evidence against it, never proof of anything.

What a p-value is, and is not

The p-value is P(data this extreme or more | H0 true). It is not the probability that H0 is true, not the probability that the result happened "by chance", and not the size or importance of the effect. A p-value of 0.03 means: if there were truly no effect, you would see a difference this large in about 3 of 100 repeated studies.

Two errors are possible. A Type I error is rejecting a true H0 (a false alarm); alpha is the rate you accept. A Type II error is missing a real effect; its rate is called beta, and 1 - beta is the power of the test. The next lessons on A/B testing come back to power.

Comparing two means: the t-test

Suppose a new page layout is shown to 60 visitors and the old one to another 60. The outcome is seconds spent on the page. We compare means with Welch's t-test, which does not assume equal variances and is the safe default.

Always report three things, not just the p-value: the difference in the original units, an interval for it, and an effect size. Cohen's d expresses the difference in standard deviations (rough guide: 0.2 small, 0.5 medium, 0.8 large; these are only conventions).

Python Welch's t-test with an interval and effect size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the first result. The true effect is 5 seconds, yet this particular sample gives p just above 0.05 and an interval that barely touches zero. "Not significant" here does not mean "no effect"; the study was simply too small to be sure. The interval says more than the verdict: the effect is plausibly anywhere from zero to about 8 seconds.

The paired test at the end shows a second design. When the same users are measured twice, ttest_rel works on the per-user differences, which removes the person-to-person noise and finds the small shift easily. Pairing is a design decision, not a trick: use it only when the data really are paired.

Note: "Statistically significant" is not "important". With a million rows, a difference of 0.01 seconds can have p < 0.001. Always ask whether the size of the effect matters for the decision.

Counts and categories: the chi-square test

When both variables are categories, you compare counts. The chi-square test of independence asks whether the distribution of one category (signed up or not) differs across the groups of another (device). It compares observed counts with the counts you would expect if the two were unrelated.

When the t-test is a poor fit: nonparametric tests

The t-test works best when the sampling distribution of the mean is roughly normal. Heavily skewed data with small samples, such as order totals, can strain it. Nonparametric tests use ranks instead of raw values, so they are robust to outliers and skew. The rank-based alternative to the two-sample t-test is the Mann-Whitney U test. It asks whether values in one group tend to be larger than in the other.

Python Chi-square on a table, Mann-Whitney on skewed data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Signup rates are 12%, 10% and 11% across devices; with p around 0.7 there is no evidence that the device matters. For the order totals, both tests flag a difference, and the rank-based p-value is smaller because the skew inflates the t-test's noise. Do not choose a test because it gives the smaller p-value; choose it from the data's shape before testing.

A rough chooser:

Question Typical test
Two group means, roughly normal Welch t-test
Same units measured twice Paired t-test
Two groups, skewed or ordinal data Mann-Whitney U
Two categorical variables Chi-square (or Fisher's exact for tiny counts)
Three or more group means One-way ANOVA (or Kruskal-Wallis)

Checking assumptions, comparing several groups and paired data

The t-test above makes two assumptions that you can actually check: the data in each group are roughly normal, and the groups have similar variances. This section shows the standard checks (Shapiro-Wilk and Levene), the test for comparing three or more means (one-way ANOVA with a post-hoc test), the paired rank test (Wilcoxon) and a variance-ratio F-test. All of them are in scipy.stats.

Assumption tests are a guide, not a gate. With big samples they flag tiny, harmless departures; with small samples they miss large ones. Look at a histogram or Q-Q plot first, use the test as a second opinion, and remember that the t-test and ANOVA are fairly forgiving when group sizes are similar and the sample is not tiny.

Normality: Shapiro-Wilk

stats.shapiro(x) tests the null hypothesis "the sample comes from a normal distribution". A small p-value is evidence against normality. It works best for samples up to a few thousand.

Equal variances: Levene

stats.levene(a, b, c, center="median") tests the null hypothesis that all groups have the same variance. The median-centred version (the Brown-Forsythe variant) is robust to non-normal data. A small p-value suggests the spread differs, so prefer Welch's t-test or Welch-type methods.

Three or more means: one-way ANOVA

Running a t-test on every pair of groups inflates the false-alarm rate, as described in the multiple-comparisons section. ANOVA (analysis of variance) first asks one overall question: "are all group means equal?" It compares the variation between group means with the variation within groups, and reports an F statistic. A significant result says "at least one mean differs", not which one. To find out which, run a post-hoc test that controls the overall error rate, such as Tukey's HSD (stats.tukey_hsd). The rank-based alternative to ANOVA is Kruskal-Wallis (stats.kruskal).

Python Normality, equal variance, ANOVA and Tukey

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output in order. If Shapiro and Levene look fine, ANOVA is appropriate. The overall p-value says the three designs are not all the same, and Tukey shows which pairs drive the difference. Report the size of each difference (here, the difference in seconds) and a confidence interval, not only p-values, as in Confidence Intervals.

A significant ANOVA does not mean every pair differs. And not significant does not mean equal: with a small sample you may simply lack the power to see a real difference.

Paired data: the Wilcoxon signed-rank test

When the same subjects are measured twice (before and after a change), the observations are paired. The paired t-test analyses the differences. If the differences are skewed or contain outliers, the Wilcoxon signed-rank test (stats.wilcoxon) uses the ranks of the differences instead. It is the paired cousin of Mann-Whitney.

Comparing two variances: the F-test

Sometimes the spread is the question: did a new process make the output more consistent? The F-test compares the ratio of two sample variances. SciPy has no one-line function for it, but the ratio and its p-value take three lines. It is very sensitive to non-normal data, so check normality first, or use Levene instead.

Python Wilcoxon on paired data and an F-test for variances

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

MANOVA in one paragraph

ANOVA handles one numeric outcome. MANOVA (multivariate analysis of variance) tests whether group means differ across several outcomes at once, such as height, weight and blood pressure for three diets, while accounting for the correlation between outcomes. It is useful when outcomes are related and you want a single overall test before looking at each one. In Python it lives in statsmodels, so the example is read-only here.

Python
# Read-only: needs the statsmodels package
from statsmodels.multivariate.manova import MANOVA

fit = MANOVA.from_formula("height + weight + bp ~ diet", data=df)
print(fit.mv_test())          # Wilks' lambda, Pillai's trace and others

A significant MANOVA is followed by separate ANOVAs, with a correction for the number of outcomes, to see which outcome moves.

One-tailed and two-tailed tests

The alternative hypothesis decides which tail of the null distribution counts as "extreme".

  • Two-tailed (two-sided) test. H1 says "different": the mean is not equal to the reference. Both very small and very large statistics count, so the 5% rejection region is split into 2.5% in each tail. Use it as the default.
  • Right-tailed (right-sided) test. H1 says "greater": the new mean is higher. Only large positive statistics count, and the whole 5% sits in the upper tail.
  • Left-tailed (left-sided) test. H1 says "smaller": for example, the new process produces fewer defects. Only large negative statistics count.

A one-sided test has more power for an effect in the expected direction, but it can never detect an effect in the other direction, so it must be chosen before you see the data, from a real reason (a drug that cannot plausibly raise the risk, a cost-saving change that only matters if costs fall). Choosing the side after peeking is p-hacking. In SciPy, the alternative= argument takes "two-sided", "greater" or "less".

Python Two-sided, right-tailed and left-tailed tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Parametric and non-parametric tests at a glance

Parametric tests assume a distribution (usually normal) and compare parameters such as means. Non-parametric tests use ranks or counts and make fewer assumptions, at some cost in power when the parametric assumptions do hold.

Question Parametric test Non-parametric alternative
One sample against a reference value one-sample t-test, z-test Wilcoxon signed-rank, sign test
Two independent groups Welch's t-test Mann-Whitney U
Two paired measurements paired t-test Wilcoxon signed-rank
Three or more groups one-way ANOVA (ANOVA and F-Tests) Kruskal-Wallis
Association of two numbers Pearson correlation Spearman, Kendall
Association of two categories (not applicable) chi-square (Chi-Square Tests)

The z-test, the z-table and the z-score

When the population standard deviation sigma is known (or the sample is large, roughly 30 or more) the test statistic for a mean is a z-score: z = (mean - mu0) / (sigma / sqrt(n)). The z-table lists the area under the standard normal curve; stats.norm.cdf(z) is the table, and stats.norm.ppf(p) reads it backwards. For two independent samples with known or large-sample variances, the two-sample z-test uses z = (mean1 - mean2) / sqrt(s1ยฒ/n1 + s2ยฒ/n2).

Solved example. A machine is set to fill bags with 500 g (known sigma = 8 g). A check of 64 bags gives a mean of 497.9 g. Is the machine under-filling (left-tailed, alpha 0.05)? z = (497.9 - 500) / (8 / 8) = -2.1, the table area to the left is 0.0179, which is below 0.05, so reject H0.

The same z-score also describes a single value: z = (x - mean) / std says how many standard deviations a point lies from the mean. A common outlier rule flags points with |z| > 3 (some analysts use 2.5 or 2 for small data). It assumes roughly normal data and is itself distorted by the outliers, so for skewed data prefer the IQR rule from Outliers and Errors.

Python A z-test by hand, the z-table and z-score outliers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Homogeneity of variances and p-values for feature selection

Tests that pool groups (the classic Student t-test and ANOVA) assume homogeneity of variance: equal spread in each group. Check it with Levene's test (shown above) or avoid the assumption with Welch's versions.

Hypothesis tests also serve feature selection: test every candidate feature against the target (t-test or ANOVA for numeric features and a class target, chi-square for categorical ones, correlation test for numeric targets) and keep those with small p-values. The advantages are speed, simplicity and a ranking that is easy to explain. The limits are real: each test looks at one feature alone, so it misses interactions and keeps redundant correlated features; testing hundreds of features produces false positives unless you correct for multiple comparisons (next section); and with big data every feature is "significant", so look at effect sizes as well. Treat it as a first filter, then confirm with a model on held-out data.

Multiple comparisons and p-hacking

Here is the trap that causes the most false findings. Every test run at alpha = 0.05 has a 5% chance of a false alarm when nothing is going on. Run 20 independent tests on pure noise and the chance of at least one "significant" result is 1 - 0.95^20, about 64%. Slice the data by country, device, week and plan, test each slice, and report the winner: that is p-hacking, also called the garden of forking paths. The result is real in the data and false in the world.

Python Twenty tests on pure noise

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The histogram is the key insight: when H0 is true, p-values are spread evenly between 0 and 1, so exactly 5% land below 0.05 by design. In this particular run, one of the 20 tests dipped below 0.05 (the smallest p was 0.020), but after correction nothing survives.

Ways to protect yourself:

  • Decide the hypotheses and the metric before looking. Write them down, even in a comment.
  • Correct for many tests. Bonferroni divides alpha by the number of tests (simple, conservative). Holm is uniformly better and equally easy. When you run hundreds of tests, controlling the false discovery rate (Benjamini-Hochberg) is more practical; scipy.stats.false_discovery_control and statsmodels.stats.multitest.multipletests implement these.
  • Label exploration honestly. Patterns found by browsing are hypotheses for the next dataset, not conclusions.
  • Confirm on fresh data. A finding that holds on data you did not use to find it deserves more trust.

Reporting a result well

A good sentence includes the estimate, its uncertainty, and the practical meaning: "Visitors on the new layout stayed 4.1 seconds longer on average (95% CI -0.02 to 8.1 s, Welch t-test p = 0.051). The data are compatible with no change and with a gain of about 8 seconds; we need a larger test before deciding." That is more useful than "not significant".

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A test gives p = 0.03. Which statement is correct?

    The p-value is a probability about the data given H0, not about H0 itself, and it says nothing about the effect size.
  2. 2. You test 20 unrelated metrics at alpha = 0.05 on pure noise. About how likely is at least one 'significant' result?

    The chance of no false alarm in 20 independent tests is 0.95^20, about 0.36, so at least one appears about 64% of the time.
  3. 3. A test is 'not significant'. What can you conclude?

    Failing to reject H0 is absence of evidence, not evidence of absence. The confidence interval shows which effect sizes remain plausible.
  4. 4. Which test suits two groups of heavily skewed values with a few extreme outliers?

    Mann-Whitney works on ranks, so extreme values have limited influence. The paired t-test needs paired data, and chi-square needs counts.
  5. 5. Why report an effect size and an interval, not just the p-value?

    A tiny effect can be highly significant in a huge sample, and a large effect can be non-significant in a small one. Size and uncertainty carry the practical meaning.

Summary

  • A test measures how surprising the data are if the null hypothesis is true; the p-value is not the probability that H0 is true.
  • Report the effect, an interval and an effect size, not only "significant" or "not significant".
  • Use Welch's t-test for two means, a paired test for paired data, Mann-Whitney for skewed data and chi-square for counts.
  • Running many tests inflates false alarms; decide hypotheses first, correct with Holm or Bonferroni, and confirm on new data.
  • "Fail to reject" means the evidence was insufficient, not that the effect is zero.

Next: A/B Testing and Experiments, where these ideas are applied to a real decision.