Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 37 ยท Power and Sample Size

Statistical Power and Error Types

Understand Type I and Type II errors, alpha, beta and statistical power in A/B testing, and see how sample size, effect size and noise drive power, with Python plots.

An experiment can fail in two opposite ways. It can claim a win that is not real, or it can miss a win that is. Statistical power is the measure of how well a test avoids the second failure. This chapter turns that idea into a practical question every test must answer before launch: how many users do we need?

The four outcomes

Every test ends in one of four situations, depending on the truth (hidden from you) and your decision:

Decide: no effect Decide: effect
Truth: no effect Correct (true negative) Type I error, false positive, probability alpha
Truth: real effect Type II error, false negative, probability beta Correct, probability power = 1 - beta
  • Alpha is the false-positive rate you accept, usually 0.05.
  • Beta is the false-negative rate; power is its complement. The convention is 80% power (beta = 0.2), and 90% for important decisions.

Alpha and beta are not symmetric on purpose: shipping a useless change usually costs less to avoid than missing a good one, and each is controlled in a different way. Alpha is a threshold you choose. Power must be planned through sample size.

What drives power?

Four quantities are tied together. Fix three and the fourth follows.

What raises power

More users Smaller standard error
Bigger effect Easier to see above noise
Less noise Lower metric variance
Higher alpha A looser bar to clear

Alpha is rarely the lever you should pull.

1 / 2

Seeing power as overlapping curves

Picture two sampling distributions of the observed lift: one centered at zero (null) and one centered at the true effect (alternative). The rejection threshold sits where the null's tail has 2.5% on each side. Power is the area of the alternative curve beyond that threshold.

Python Null vs alternative distributions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The blue shaded area is the probability that an experiment with a real +1 point effect crosses the significance line. Increase n and both curves get narrower, so the overlap shrinks and power rises.

Power as a function of sample size

We can compute power exactly for the z-test and watch it climb with sample size.

Python Power curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice two lessons. Doubling the effect needs far fewer users (the +2 point column reaches high power much sooner), and extra users pay off with diminishing returns once power is already high. Required sample size scales with 1 / effect^2: halving the effect you want to detect needs about four times the users.

Why underpowered tests are dangerous

Low power does not only mean missed wins. Two further problems:

  1. Most "significant" results are inflated. In an underpowered test, only unusually lucky samples cross the line, so a significant estimate tends to overstate the true effect (the winner's curse).
  2. Null results are uninformative. "No significant difference" from a tiny test tells you almost nothing; it is not evidence of no effect.
Python Winner's curse in an underpowered test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "winners" report an average lift well above the truth. If you ship based on those, you will be disappointed. Adequate power is also a defense against exaggerated results.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a Type II error?

    A Type II error (false negative) is missing a true effect. Its probability is beta, and power is 1 - beta.
  2. 2. Which change increases power, other things equal?

    A larger effect stands out more from noise. More users and less noise also help, while a smaller alpha lowers power.
  3. 3. To detect an effect half as large with the same power, roughly how many more users do you need?

    Required sample size is proportional to 1 / effect squared, so halving the effect quadruples the sample.
  4. 4. What is the winner's curse?

    Only unusually large observed lifts cross the significance line when power is low, so the published winners are biased upward.

Summary

  • Type I error (false positive, alpha) and Type II error (false negative, beta) are the two ways a test fails; power = 1 - beta.
  • Power depends on sample size, effect size, noise and alpha; sample size scales with 1 / effect squared.
  • Underpowered tests miss wins, give uninformative nulls and exaggerate the wins they do find.
  • Plan for 80-90% power before launch.

Next: Sample Size for Conversion Rates