Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 37 ยท Variance Reduction

Stratification and Post-Stratification

Reduce A/B test variance by comparing like with like: stratified randomization, post-stratification weights, and a simulation showing the narrower estimates, in runnable Python.

CUPED adjusts for a numeric covariate. Often the helpful variable is a category: country, device type, new versus returning user, plan tier. If those groups differ a lot in the metric (desktop users spend three times what mobile users do), then random bad luck in how many desktop users land in each variant adds noise. Stratification removes that source of noise by comparing like with like.

The idea: split users into strata, estimate the treatment effect inside each stratum, and combine the stratum effects using each stratum's population share as weight.

Two ways to stratify

  • Pre-stratification (blocked randomization). At assignment time, randomize separately inside each stratum so each variant gets the same mix. It guarantees balance but needs the stratum known at assignment and supported by your platform.
  • Post-stratification. Randomize as usual, then at analysis time compute the effect per stratum and re-weight to the true population proportions. It works with any platform and gets almost all of the variance reduction when samples are large.

Post-stratification

Split into strata Country, device, tier
Effect per stratum Treatment minus control
Weight by share Population proportions
Less noise Group mix luck is removed

Cheap, safe, easy to explain

1 / 2

The estimator

With strata s, stratum weight w_s (its share of all users), and group means mean_T(s) and mean_C(s):

TEXT
effect = sum over s of  w_s * (mean_T(s) - mean_C(s))
Python Plain versus post-stratified estimate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In this single run the two estimates nearly coincide (the lift happens to be a bit above 1.5 in both). The benefit shows up over many experiments, as the next example shows: post-stratification ignores how many desktop users happened to land in each group.

Measuring the variance reduction

One experiment cannot show reduced variance. Repeat it 400 times and compare the spread.

Python Spread of the estimate over 400 experiments

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both are unbiased; the post-stratified spread is clearly smaller. The size of the gain equals the share of variance explained by the strata, here the large differences between device averages.

Practical advice

  • Choose strata that predict the metric and were defined before the test (device, country, platform, plan). Strata that do not predict it give no gain.
  • Keep strata large enough. With a stratum of 20 users in one arm, its mean is unstable; merge small strata.
  • Use population weights, not per-arm sample shares. Use the overall share of each stratum (or a known external share), so both arms are re-weighted to the same mix.
  • Never stratify on post-assignment variables (for example, "users who clicked"). That variable can be changed by the treatment.
  • Combine with CUPED. Stratified CUPED fits theta separately per stratum or adds stratum indicators in a regression.
  • A regression gives nearly the same answer. Fitting y ~ group + C(device) is close to post-stratification, and an interaction model with centered covariates reproduces it.

Check for balance in your A/A tests as well: if a stratified A/A test keeps showing a significant effect, you probably have an assignment bug, not a statistics problem (see sample ratio mismatch).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does stratification reduce variance?

    Comparing within strata and re-weighting removes the variance that comes from random imbalance in group composition.
  2. 2. What are the correct weights in post-stratification?

    Weights are population shares, applied identically to both arms.
  3. 3. Which variable is a valid stratifier?

    Stratifiers must be fixed before assignment so the treatment cannot change them.
  4. 4. A stratifier unrelated to the metric will...

    The estimator stays valid but gains nothing if strata do not predict the metric.

Summary

  • Stratification compares like with like: effect per stratum, combined with population weights.
  • Pre-stratification balances at assignment; post-stratification re-weights at analysis.
  • Both are unbiased and reduce variance in proportion to how well the strata predict the metric.
  • Use pre-assignment, sizeable strata; combine with CUPED when you have numeric covariates.

Next: The Delta Method for Ratio Metrics