Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 57 ยท Applied Statistics

Nonparametric Tests

When data are skewed, ordinal or tiny, use rank-based tests: Mann-Whitney U, Wilcoxon signed-rank, Kruskal-Wallis and sign test, with SciPy examples.

The t-test, ANOVA and Pearson correlation are parametric: they assume the data (or errors) follow a particular distribution, usually normal. Real data often breaks that: income is heavily skewed, a satisfaction survey gives ratings from 1 to 5, an experiment has only eight subjects. Nonparametric (distribution-free) tests make far weaker assumptions. Most of them work on the ranks of the data instead of the raw values, which also makes them robust to outliers.

When to choose a nonparametric test

  • The data are ordinal (ranked categories such as "poor, fair, good, excellent").
  • The sample is small and clearly not normal, so you cannot lean on the central limit theorem.
  • The data have strong outliers or skew that you cannot or should not remove.
  • You care about the median or about "tends to be larger" more than the mean.

The price: when the data really are normal, a nonparametric test has slightly less power (about 95% relative efficiency for the Wilcoxon family), so you need a few more observations to detect the same effect. For large samples the difference rarely matters.

Which test replaces which

Situation Parametric test Nonparametric alternative
Two independent groups independent t-test Mann-Whitney U (Wilcoxon rank-sum)
Paired / before-after paired t-test Wilcoxon signed-rank (or sign test)
3+ independent groups one-way ANOVA Kruskal-Wallis H
Repeated measures, 3+ conditions repeated-measures ANOVA Friedman test
Correlation Pearson Spearman or Kendall
Distribution shape comparison none Kolmogorov-Smirnov

Mann-Whitney U: two independent groups

Pool all observations, rank them from smallest to largest (ties share the average rank), and add up the ranks within each group. If the groups come from the same distribution, the rank sums should be similar. The null hypothesis is that a random value from group A is equally likely to be larger or smaller than a random value from group B. Strictly it compares whole distributions; with similarly shaped groups it amounts to a comparison of medians.

Python Mann-Whitney U vs t-test on skewed data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single 9.5-second outlier inflates site A's mean and variance and hides the difference from the t-test, while the rank-based test is barely affected by it and detects that B is consistently slower.

Wilcoxon signed-rank and the sign test: paired data

For paired measurements (the same person before and after), compute the differences. The sign test uses only the sign of each difference: if there is no effect, positives and negatives are equally likely, so the number of positives follows a Binomial(n, 0.5). It is extremely assumption-light but wastes information. The Wilcoxon signed-rank test also uses the size: rank the absolute differences and sum the ranks of the positive ones. It assumes the differences are roughly symmetric about their median.

Python Paired data: sign test and Wilcoxon

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Kruskal-Wallis: three or more groups

Kruskal-Wallis is the rank-based version of one-way ANOVA. It ranks all observations together and checks whether the average ranks differ among groups by more than chance would allow. A significant result says some group differs, not which. Follow up with pairwise comparisons such as Dunn's test or Mann-Whitney tests with a multiple-comparison correction (see Multiple Testing and Effect Sizes).

Python Kruskal-Wallis on three teaching methods

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Goodness of fit: Kolmogorov-Smirnov

The KS test compares a sample's empirical cumulative distribution with a reference distribution (one-sample) or with another sample's (two-sample). The statistic D is the largest vertical gap between the two CDFs. It is sensitive to differences in location, spread and shape, but weak against subtle tail differences. When parameters are estimated from the same data, the standard p-values are too generous; use a Lilliefors-type correction.

Python KS test: normal or not?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical advice

  • State what you test: Mann-Whitney does not test means. Report medians (with the quartiles) and an effect size such as the rank-biserial correlation or the probability of superiority U/(nโ‚nโ‚‚).
  • Ties reduce accuracy; SciPy applies corrections.
  • Normality tests (Shapiro-Wilk) have little power with small samples and flag trivial deviations with huge ones. Look at a histogram or Q-Q plot, and think about how the data arose, rather than letting a p-value choose the test.
  • A permutation test (Bootstrap and Permutation Tests) is another assumption-light option and works for any statistic.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which test is the nonparametric alternative to the independent two-sample t-test?

    Mann-Whitney U (also called the Wilcoxon rank-sum test) compares two independent groups using ranks.
  2. 2. What is the main reason nonparametric tests are robust to outliers?

    Ranking replaces each value with its position, so a huge outlier has the same effect as a value just slightly above the rest.
  3. 3. You have before-and-after scores for 12 patients and the differences are strongly skewed. Which test fits?

    Paired data call for a paired method, and the rank-based Wilcoxon signed-rank or the sign test avoids the normality assumption.
  4. 4. Kruskal-Wallis returns p = 0.002 for four groups. What does this tell you?

    It is an omnibus test like ANOVA: significance says some difference exists, not where.

Summary

  • Nonparametric tests use ranks or signs and assume little about the distribution, so they suit ordinal, skewed, small or outlier-prone data.
  • Mann-Whitney (2 independent groups), Wilcoxon signed-rank / sign test (paired), Kruskal-Wallis (3+ groups), Friedman (repeated measures), KS (distribution shape).
  • They lose a little power on truly normal data but can gain a lot when assumptions fail.
  • Report medians and an effect size, not only a p-value.

Next: A/B Testing