Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 37 ยท Variance Reduction

CUPED: Using Pre-Experiment Data

Learn CUPED: adjust A/B test metrics with pre-experiment covariates to cut variance by 30-60%, derive theta, check validity and compare with regression adjustment in Python.

Some of the noise in your metric is not noise at all: it is who the users are. A heavy buyer last month will probably be a heavy buyer during your test, no matter which variant they see. If you can measure that predictable part before the experiment starts, you can subtract it out and compare what is left. This is CUPED (Controlled-experiment Using Pre-Experiment Data), the most widely used variance-reduction technique in industry.

The key promise: because the covariate is measured before randomization, it cannot be affected by the treatment. Subtracting it leaves the effect estimate unbiased while shrinking its variance.

The formula

Let Y be the metric during the experiment and X the same metric (or any predictive covariate) measured in a pre-experiment window. Define:

TEXT
Y_cuped = Y - theta * (X - mean(X))
theta   = cov(X, Y) / var(X)

Estimate theta on the pooled data from both groups. The new variance is:

TEXT
var(Y_cuped) = var(Y) * (1 - rho^2)

where rho is the correlation between X and Y. A correlation of 0.7 removes about 51% of the variance; 0.5 removes 25%. The better the pre-period predicts the experiment-period, the larger the gain.

CUPED in four steps

Collect X Metric before the test
Fit theta cov(X, Y) / var(X), pooled
Adjust Y Y - theta (X - mean X)
Compare groups Same t-test, smaller noise

Effect unbiased, variance smaller

1 / 2

A runnable example

We simulate users with a persistent "spending level", add a true treatment lift of 1.0 dollar, and compare the plain and CUPED analyses.

Python Plain difference versus CUPED

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both analyses estimate a lift near 1.0, but the CUPED standard error is much smaller, so the p-value becomes decisive. The variance ratio printed at the end should be close to 1 - rho^2.

Is it really unbiased? Check across many experiments

Run the same experiment many times and compare the average estimate and the spread of both methods.

Python Repeating the experiment 300 times

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The averages match the truth for both methods (unbiased), but the CUPED estimates are much tighter. That tighter spread is what shortens your experiments.

Choosing the covariate

The best covariate is the same metric from a pre-experiment window of similar length, often 1-4 weeks. Other options:

  • Related metrics (sessions, page views) when the main metric did not exist before.
  • Several covariates at once: fit a linear regression of Y on X1, X2, ... and use the residuals. This is the same as regression adjustment and is what many platforms call CUPAC or ML-based CUPED when a model produces the prediction.
  • Never use anything measured after assignment. A post-treatment covariate can itself be affected by the variant and will bias the result.

Practical details

  • New users have no pre-period. Give them X = mean(X) (zero adjustment), or use a covariate that exists at signup, such as country or device.
  • Estimate theta on pooled data, not separately per group, and compute it once.
  • Correct standard errors must treat theta as estimated. With thousands of users the difference is negligible, which is why the simple t-test on Y_cuped is standard practice.
  • Weak correlation, weak gain. If rho is around 0.1, CUPED removes only 1% of variance. Check the correlation before bothering.
  • Report the original metric in dashboards. CUPED adjusts the estimate of the lift, not the raw averages.

CUPED pairs well with other ideas in this chapter: stratification when covariates are categorical, and the delta method when the metric is a ratio.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is CUPED safe from bias?

    A pre-experiment covariate is independent of the assignment, so adjusting for it does not change the expected effect.
  2. 2. If the correlation between pre and post metric is 0.8, roughly how much variance does CUPED remove?

    Variance falls by a factor of 1 - rho^2 = 1 - 0.64 = 0.36, a 64% reduction.
  3. 3. Which covariate would be a mistake?

    A covariate measured after assignment can be changed by the variant and bias the estimate.
  4. 4. How should theta be estimated?

    Pooled estimation keeps the adjustment symmetric between the groups.

Summary

  • CUPED subtracts the predictable part of the metric using a pre-experiment covariate.
  • The adjusted variance is var(Y) * (1 - rho^2); strong correlation means large savings.
  • It leaves the treatment effect unbiased because the covariate predates randomization.
  • Use pooled theta, handle users without history, and never adjust for post-assignment variables.
  • Multiple covariates lead to regression adjustment and model-based variants.

Next: Stratification and Post-Stratification