Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Pitfalls and Ethics

Winner's Curse and Simpson's Paradox

Why shipped A/B test wins shrink after launch, and how an uneven traffic mix can flip a result: simulate the winner's curse and Simpson's paradox in Python and learn the remedies.

Two statistical traps catch even careful teams. The winner's curse makes the experiments you ship look better than they truly are. Simpson's paradox can make a variant look worse overall while it is better in every segment. Both come from the same root: selecting or aggregating data in a way that hides how the numbers were produced.

The winner's curse

Suppose your team runs 100 experiments a quarter. Most changes have a tiny true effect, some are slightly positive, some slightly negative. You ship the ones with significant positive results. Among those, the observed lift is partly real effect and partly luck in the right direction. By choosing winners you also choose lucky noise. The estimate of the lift for selected winners is biased upward, and the effect you actually see after launch is smaller.

Python Observed lift versus true lift of winners

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The observed lift of the winners is noticeably larger than their true lift, and a few "winners" are actually harmful. The more underpowered your tests, the stronger the effect.

Why wins shrink

Select winners Only significant ones ship
Luck is included Noise that favored B
After launch Luck does not repeat

Selection creates bias

1 / 2

Remedies:

  • Power your tests properly. Bigger samples mean less noise to select on.
  • Shrink estimates. Empirical Bayes uses the history of past experiments to estimate the distribution of true effects and pulls each new estimate toward it.
  • Replicate big surprises. Re-run, or keep a holdback group after launch, to measure the real effect.
  • Do not add up the lifts of shipped tests to forecast revenue. The sum of winners overshoots reality.

Simpson's paradox

Simpson's paradox means a trend that appears in each segment reverses when segments are combined. In experiments it appears when the mix of segments differs between variants, for example if traffic allocation changes during the test or if the variants are not randomized equally across segments.

Example: a new checkout (B) is better for both mobile and desktop. But B got most of its traffic from mobile users (low converters), while A got most of its traffic from desktop users (high converters). The overall rates reverse.

Python B wins in every segment, A wins overall

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Inside each segment B converts better (11% vs 10% on desktop, 4.25% vs 4% on mobile), yet overall A shows 8.8% and B only 5.6%, because B's users are mostly mobile.

What to do about it

In a properly randomized experiment with a fixed allocation, both variants receive the same segment mix (apart from chance), and the paradox does not arise. It shows up when:

  • The allocation changed during the test (for example, ramping B from 10% to 50%). Weeks with different traffic mixes or seasonality are then unevenly weighted between arms. Either analyze each ramp stage separately and combine with weights, or discard the ramp period.
  • A bug skews assignment toward certain segments (check the sample ratio and segment balance).
  • You compare across experiments or time periods rather than within a randomized split.
Python Fix: compare within segments and re-weight

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Re-weighting both variants to the same segment mix (the same idea as post-stratification) restores the correct ordering: B above A.

When an overall result contradicts the segment results, do not pick a side. Ask: "Is the segment mix the same in both arms?" If not, trust the standardized comparison.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do shipped A/B winners often deliver less than measured?

    The tests that cross the threshold are partly those where noise helped, so their estimates are biased upward.
  2. 2. Which practice reduces the winner's curse?

    Less noise and shrinkage toward the prior distribution of effects reduce the selection bias.
  3. 3. Simpson's paradox in an experiment typically requires...

    When the mix of segments differs between arms, overall rates can reverse relative to within-segment rates.
  4. 4. How do you resolve the paradox?

    Standardizing to a common segment mix compares like with like.

Summary

  • Selecting significant winners overstates their effect; expect shipped wins to shrink.
  • Power tests well, shrink estimates and replicate or hold back to measure the real effect.
  • Simpson's paradox appears when segment mixes differ between arms, often after allocation changes or bugs.
  • Compare within segments and standardize the mix to resolve it.

Next: Ethics, Privacy and Responsible Experimentation