Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 37 ยท Variance Reduction

Why Variance Reduction Matters

Understand how metric variance drives A/B test sample size and duration, and try capping, winsorizing and metric choice to shrink noise, with runnable examples.

Every A/B test is a signal-to-noise problem. The signal is the true effect of your change. The noise is how much your metric jumps around between users. If the noise is large, you need many users before the signal can be told apart from luck. If you can cut the noise, you can reach a decision faster, detect smaller effects, or both.

This chapter is about cutting noise without cheating: methods that make the same data more informative while keeping the test valid.

Variance sets the sample size

For a two-group comparison of means, the sample size per group needed for 80% power at a 5% significance level is approximately:

TEXT
n = 16 * sigma^2 / delta^2

where sigma is the standard deviation of the metric and delta is the smallest effect you want to detect. Two consequences follow:

  • Halve the variance, halve the sample size. Reducing sigma^2 by 50% means you need half the users.
  • Detecting half the effect costs four times the users, because delta is squared.
Python Sample size versus variance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 40% variance reduction cuts the required traffic by 40%. For a site with modest traffic, that can turn a 6-week test into a 3-week test.

Where variance comes from

Heavy tails A few huge spenders
User differences Heavy users vs casual
Time effects Weekdays, seasons
Pure luck What the user did that day

Some can be removed, some cannot

1 / 2

The simplest lever: choose a less noisy metric

Before any clever statistics, ask whether you are measuring the right thing.

  • Binary beats continuous. "Did the user purchase?" has far lower relative noise than "how many dollars did the user spend?".
  • Closer to the change. A button color affects clicks directly; revenue sits many steps downstream and is noisier.
  • Shorter windows can help if the effect shows up early, but beware of novelty effects (novelty and primacy).
  • Per-user over per-event. Randomizing by user but analyzing per page view overstates your sample size; use the user as the unit of analysis.

Capping and winsorizing

Revenue per user is heavy tailed: most users spend nothing, a few spend hundreds. One whale in group B can swing the average and the variance. Winsorizing replaces values above a chosen percentile (often the 99th) by that percentile. It reduces variance, at the cost of some bias (the capped metric is a different metric, so define it before the test).

Python Capping a heavy-tailed revenue metric

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Capping at the 99.9th percentile removes about a third of the variance and barely changes the metric. Capping at the 99th percentile removes almost 80% of the variance, but the mean drops by about 30%: because only 6% of users buy, the 99th percentile sits among ordinary buyers and the cap now cuts real revenue. The cap is a trade-off between noise and bias, and the capped metric is a different metric from raw revenue. Rules of thumb:

  • Pick the cap from pooled data across both groups, before looking at which group wins.
  • Document it: "capped revenue per user at the 99th percentile".
  • Report both the capped result (for the decision) and the uncapped one (for transparency) if the business cares about whales.

A log transform is a different question

Taking log(1 + revenue) also tames the tail, but a test on the log metric compares geometric-like averages, not total revenue. It answers "did typical spending rise?", not "did the sum of revenue rise?". Use it only when that is truly the question.

Python Capping versus log: what do they measure?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Here the raw difference is driven by a handful of whales; the capped and log versions say there is little change for typical users. Which one is right depends on the business question, and that is exactly why the choice must be made before the test.

Never choose the cap, the transformation or the outlier rule after seeing which choice makes the result significant. That is a form of the multiple comparisons problem in disguise.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. If you cut a metric's variance by 50%, what happens to the required sample size per group?

    Sample size is proportional to variance, so half the variance needs about half the users.
  2. 2. You want to detect an effect half as large as before. How does the sample size change?

    n is proportional to 1 / delta^2, so halving delta quadruples n.
  3. 3. Which is the safest way to choose a winsorizing cap?

    A pre-registered, pooled cap avoids data-dependent choices that bias the decision.
  4. 4. A test on log(1 + revenue) answers which question?

    The log shrinks big values, so it reflects typical spending rather than the sum.

Summary

  • Required sample size scales with metric variance and with 1 / effect squared.
  • Choose metrics that are close to the change, with the user as unit of analysis.
  • Winsorizing at a pooled percentile trades variance for bias (a high percentile keeps the bias small); decide before the test.
  • Log transforms change the question, not just the noise.
  • The next lessons add model-based variance reduction: CUPED, stratification and the delta method.

Next: CUPED: Using Pre-Experiment Data