Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 37 ยท Potential Outcomes

Randomization Inference and the Bootstrap

Quantify uncertainty in causal estimates without formulas: Fisher's permutation test for the sharp null and bootstrap confidence intervals for the ATE, with Python.

An effect estimate is useless without a statement of uncertainty: how different might it be in another sample, or by luck of the assignment? Two simple, assumption-light tools cover most cases: randomization (permutation) inference, which tests whether any effect exists, and the bootstrap, which gives confidence intervals for effect sizes.

Fisher's sharp null and permutation tests

Ronald Fisher's idea: assume the sharp null hypothesis that the treatment has no effect on any unit, Y_i(1) = Y_i(0) for all i. Then each unit's outcome would have been the same whichever group it was in, so we know all potential outcomes. We can reshuffle the treatment labels at random, recompute the statistic each time, and see how unusual the real result is.

Steps:

  1. Compute the observed statistic (difference in means).
  2. Shuffle the treatment labels many times, recompute each time.
  3. The p-value is the share of shuffles with a statistic at least as extreme.
Python Randomization test for a small experiment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The test assumes only that the assignment was random, not that outcomes are normal. It is exact in small samples and ideal for small trials.

flowchart LR
  A["Real labels"] --> B["observed statistic"]
  C["Shuffle labels many times"] --> D["null distribution"]
  B --> E["p-value: how extreme?"]
  D --> E

The shuffled statistics show what chance alone produces when there is no effect.

The bootstrap for the effect size

The permutation test answers "is there any effect?". For a plausible range of the effect, use the bootstrap: resample the observed data with replacement many times and recompute the estimate each time. The spread of those estimates approximates the sampling variability.

Python Bootstrap confidence interval for the ATE

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Resample rows (units), keeping each unit's treatment and outcome together. The percentile interval has no formula and works for complicated estimators.

Bootstrapping a multi-step estimator

The bootstrap really shines when the estimator has several steps (fit a model, then predict, then average) where standard errors are awkward. Repeat the whole pipeline on every resample.

Python Bootstrap for a regression-adjusted ATE

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same loop would work if the estimator were matching or inverse probability weighting. For matching estimators the plain bootstrap can be invalid, so use specialized variance estimates there.

Which tool when?

Goal Tool Notes
Is there any effect (small randomized trial)? Permutation test Exact, needs random assignment
How large and how uncertain is the effect? Bootstrap interval or robust standard error Works for complex estimators
Many groups or clusters Resample whole clusters Respect the data structure
Observational data Same tools, plus the assumptions of the design Intervals reflect sampling noise, not bias

A narrow interval does not mean the answer is right. Confidence intervals cover sampling noise only; confounding bias is not in them. Always pair uncertainty with a discussion of assumptions.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Fisher's sharp null hypothesis states that:

    Under it every unit's outcome is the same under both treatments, so labels can be reshuffled.
  2. 2. In a permutation test the p-value is:

    It measures how surprising the observed result is under random relabeling.
  3. 3. When bootstrapping a treatment effect you should resample:

    Each unit's treatment, covariates and outcome stay together.
  4. 4. A narrow bootstrap confidence interval in an observational study shows:

    Intervals quantify noise, not violations of the assumptions.

Summary

  • Permutation tests reshuffle treatment labels under the sharp null and need only random assignment.
  • The bootstrap resamples units to get intervals for any estimator, including multi-step ones.
  • Resample whole units, or clusters when data are clustered.
  • Uncertainty intervals never account for bias from confounding.

Next: Causal DAGs