Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 37 ยท Hypothesis Tests and Metrics

Bootstrap and Mann-Whitney Tests

Analyze A/B tests without strong assumptions. Compute bootstrap confidence intervals, run permutation tests and use the Mann-Whitney U test for skewed metrics in Python.

The z-test and t-test lean on formulas and a normal approximation. Sometimes you want fewer assumptions: the metric is a median, a percentile or a ratio with no tidy formula, or the sample is small and skewed. Two flexible families help: resampling methods (the bootstrap and permutation tests) and rank tests (Mann-Whitney U).

The bootstrap

The idea: your sample is your best picture of the population. To see how an estimate would vary across experiments, resample your own data with replacement many times, recompute the statistic each time, and look at the spread of those recomputed values.

flowchart LR
    D["Observed sample<br/>n users"] --> R1["resample 1<br/>(with replacement)"]
    D --> R2["resample 2"]
    D --> R3["... 2000 times"]
    R1 --> S1["statistic 1"]
    R2 --> S2["statistic 2"]
    R3 --> S3["statistic k"]
    S1 --> CI["percentile interval<br/>2.5% to 97.5%"]
    S2 --> CI
    S3 --> CI

The distribution of recomputed statistics stands in for the sampling distribution.

For an A/B test, resample each arm separately, compute mean_B - mean_A each time, and take the 2.5th and 97.5th percentiles as a 95% interval.

Python Bootstrap CI for the difference in means

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a large sample the two intervals are nearly the same. The bootstrap shines when no formula exists, for example the difference in medians or the 90th percentile of latency:

Python Bootstrap for a percentile

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Permutation tests

A permutation test gives a p-value without any distributional formula. Under the null, the variant labels are arbitrary. So shuffle the labels many times, recompute the difference each time, and see how often a shuffled difference is at least as extreme as the real one.

Python Permutation test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Adding 1 to numerator and denominator avoids reporting an impossible p-value of exactly zero.

The Mann-Whitney U test

The Mann-Whitney U test (also the Wilcoxon rank-sum test) replaces values by their ranks and compares the rank sums of the two groups. Because only order matters, it is insensitive to outliers and works on ordinal data.

Be careful about what it tests. It asks whether a random user from B tends to have a larger value than a random user from A (the probability of superiority). That is not the same as a difference in means: two groups can have equal means and still differ in rank, and vice versa. When your decision depends on the average (revenue per user, which determines total revenue), the t-test or bootstrap of the mean is the matching tool. Use Mann-Whitney when the typical user experience is the question, or for ratings and other ordinal scores.

Python Mann-Whitney vs t-test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A probability of superiority of 0.5 means no difference; above 0.5 favors B.

Choosing between methods

Situation Good choice
Large sample, mean of any metric Welch t-test (CLT makes it valid)
Median, percentile, odds ratio, no formula Bootstrap CI
Small sample, skewed, want an exact-ish p-value Permutation test
Ordinal ratings, "which tends to be higher?" Mann-Whitney U

Resampling is not a free lunch: it assumes users are independent and identically behaved, and it cannot fix a biased design. It also costs computation (thousands of resamples), which is trivial for a single experiment and heavy for a platform running thousands.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a bootstrap, how do you create each resample?

    Sampling with replacement mimics drawing a fresh sample, and the spread of the recomputed statistic approximates its sampling variability.
  2. 2. What does a permutation test shuffle?

    If the variant had no effect, any assignment of labels is equally likely, so shuffled labels generate the null distribution.
  3. 3. What does the Mann-Whitney test compare?

    It is based on ranks and tests probability of superiority, which is different from a difference in means.
  4. 4. When is the bootstrap especially useful?

    The bootstrap works for almost any statistic without needing an analytical formula for its standard error.

Summary

  • The bootstrap resamples with replacement to get confidence intervals for any statistic, including medians and percentiles.
  • A permutation test shuffles labels to get a p-value without distributional assumptions.
  • Mann-Whitney U tests rank differences (probability of superiority), not differences in means.
  • For large samples, Welch's t-test of means is still the workhorse; use resampling when formulas are missing.

Next: Multiple Comparisons