Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 57 ยท Bayesian Statistics and Simulation

Bootstrap and Permutation Tests

Learn resampling: the bootstrap for confidence intervals of any statistic and permutation tests for comparing groups, without relying on normality assumptions.

Classical formulas for confidence intervals and p-values assume a particular shape: normal data, large samples, a statistic like the mean. But what about the median, a ratio, a correlation, or an unusual statistic of your own invention? There may be no tidy formula. Resampling methods use the computer instead: they reuse your one sample, shuffled in clever ways, to learn how much a statistic varies. The two workhorses are the bootstrap (for confidence intervals) and the permutation test (for p-values).

The bootstrap idea

You want to know how a statistic would vary if you could draw many new samples from the population. You can't, but your sample is your best picture of the population. So treat the sample as the population and resample from it with replacement, drawing n values to make a new "bootstrap sample" of the same size. Compute the statistic on each. Repeat thousands of times. The spread of those bootstrap statistics approximates the sampling distribution.

flowchart TD
  A["Original sample of n values"] --> B["Draw n values with replacement"]
  B --> C["Compute the statistic"]
  C --> D["Repeat 5,000+ times"]
  D --> E["Histogram of statistics = bootstrap distribution"]
  E --> F["Percentiles give a confidence interval"]

"With replacement" matters: each value can appear several times or not at all in a resample, which creates the variation.

Python Bootstrap confidence interval for the median

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The percentile interval takes the 2.5th and 97.5th percentiles of the bootstrap statistics. Improved versions (BCa) correct bias and skew; SciPy provides them in scipy.stats.bootstrap.

Bootstrap for any statistic

The same loop works for a correlation, a trimmed mean, a ratio of means or a regression slope. Resample rows (keeping each pair (x, y) together) when the data are paired.

Python Bootstrap for a correlation, using SciPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When the bootstrap struggles

  • Very small samples: you cannot learn much about the population from five values.
  • Extreme statistics such as the maximum or minimum, which depend on the tails the sample may not contain.
  • Dependent data (time series, clusters): resample whole blocks or clusters, not single values.
  • It does not fix a biased sample: a bootstrap of a non-random sample is still non-random.

Permutation tests

A permutation test answers "is the difference between groups bigger than random labeling would produce?" without assuming normality. The logic:

  1. State the null hypothesis: the group labels do not matter.
  2. Compute the observed test statistic, for example the difference in group means.
  3. Under the null, any shuffling of the labels is equally likely. Shuffle the labels many times; each time recompute the statistic.
  4. The p-value is the proportion of shuffles with a statistic at least as extreme as the observed one.

It is the same logic as any hypothesis test from the earlier chapters, with the null distribution built by shuffling rather than looked up in a table.

Python Permutation test for a difference in means

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because the p-value is an estimate from B shuffles, report it as at least 1/B rather than exactly 0; a common formula is (count + 1) / (B + 1). Compare the result with the t-test from the hypothesis-testing chapter: for well-behaved data they agree closely, but the permutation test remains valid when the data are skewed or the statistic is unusual.

Python Permutation test for the median difference

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Bootstrap or permutation?

Bootstrap Permutation test
Main purpose uncertainty (standard error, confidence interval) significance (p-value) for a null of "no difference"
Resamples with replacement from the sample shuffles of labels, no replacement
Null hypothesis needed? no yes, labels are exchangeable
Typical use median, ratio, correlation CI comparing groups, correlation test

Both are cheap today, so use them freely, with at least 5,000 to 10,000 resamples. They also help you understand classical tests: the sampling distribution stops being a formula and becomes something you can watch.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In the bootstrap, why do we sample with replacement?

    Sampling with replacement lets values repeat or be left out, which creates the sample-to-sample variation we want to measure. Without replacement, each resample would equal the original.
  2. 2. What does a permutation test shuffle?

    Under the null that group membership does not matter, labels are exchangeable, so shuffling them builds the null distribution of the statistic.
  3. 3. Which problem is the bootstrap NOT able to fix?

    Resampling only reuses the data you have. If the sample is unrepresentative, the bootstrap reproduces that bias.
  4. 4. A permutation test with 10,000 shuffles finds 0 shuffles as extreme as observed. How should you report the p-value?

    A Monte Carlo p-value cannot be exactly zero. Report it as smaller than about 1/(B+1).

Summary

  • The bootstrap resamples your data with replacement to estimate the sampling variability of any statistic; percentiles of the results form a confidence interval.
  • Permutation tests shuffle labels to build the null distribution and give a p-value without distributional assumptions.
  • Both rely on computing power rather than formulas, and both inherit the quality of your sample.
  • Resample pairs or clusters, not single values, when the data are paired or dependent.

Next: Nonparametric Tests