Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 37 ยท Hypothesis Tests and Metrics

The Two-Proportion Z-Test

Analyze conversion-rate A/B tests with the two-proportion z-test. Learn the formula, pooled standard error, p-values, and check your work against statsmodels and SciPy.

The most common A/B test question is: "Did the conversion rate differ between A and B?" The standard tool is the two-proportion z-test. It compares two rates and tells you how surprising their gap is if the change actually did nothing.

You have already used it as a black box in earlier lessons. Here you build it step by step so you know what every number means.

The idea

We observe x_A conversions out of n_A users and x_B out of n_B. The estimated rates are p_A = x_A / n_A and p_B = x_B / n_B, and the observed lift is d = p_B - p_A.

Under the null hypothesis, both arms share one true rate. Our best estimate of that shared rate is the pooled rate:

p_pool = (x_A + x_B) / (n_A + n_B)

The standard error of the difference under the null is:

SE = sqrt( p_pool * (1 - p_pool) * (1/n_A + 1/n_B) )

The test statistic is how many standard errors the observed lift sits from zero:

z = d / SE

For large samples z follows a standard normal distribution under the null, so the two-sided p-value is 2 * P(Z > |z|).

flowchart LR
    D["observed data<br/>x_A/n_A, x_B/n_B"] --> P["pooled rate"]
    P --> SE["standard error"]
    D --> L["lift d = p_B - p_A"]
    L --> Z["z = d / SE"]
    SE --> Z
    Z --> PV["p-value<br/>2 * P(Z > |z|)"]

Pooling is used for the test because the null says the rates are equal.

Step by step in code

Python Z-test from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a z of about 2.6 and a p-value under 0.01, a gap this large would be rare if the change did nothing. We reject the null and call the lift statistically significant.

Check against a library

In real work you would use a tested library rather than hand-rolled math. statsmodels has proportions_ztest; here we confirm it matches our formula.

Python statsmodels agrees

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The statistic and p-value match the manual calculation. Note the order: the first count is the group whose rate is expected to be larger in the "larger" alternative.

When is the z-test valid?

The normal approximation needs enough data in each cell. A standard check is that all four of n*p and n*(1-p) (for each arm, using the pooled rate) are at least 5, or better 10. With a 10% conversion rate you need at least 100 users per arm to reach n*p = 10. For tiny samples or very rare events use Fisher's exact test.

Other assumptions:

  • Independent users: each user counted once, no repeat rows.
  • Fixed sample size: you analyze once at the planned end. Repeated looks need sequential methods (The Peeking Problem).
  • Random assignment: and no sample ratio mismatch.

Pooled or unpooled?

The test above uses the pooled rate because the null says the two rates are equal. When you instead want a confidence interval for the lift, you do not assume equality, so you use the unpooled standard error sqrt(p_A(1-p_A)/n_A + p_B(1-p_B)/n_B). The two give slightly different numbers; the test and its p-value use pooled, the interval uses unpooled. See Confidence Intervals for Lift.

Statistical vs practical significance

A p-value below 0.05 only says the effect is unlikely to be zero. With millions of users, a lift of 0.01 percentage points can be "significant" yet worthless. Always look at the size of the effect and compare it with what is worth shipping.

Python Significant is not the same as important

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first test has a tiny, probably useless lift that is "significant"; the second has a big lift that the data cannot yet distinguish from noise. Both facts matter for the decision.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the z-test use the pooled conversion rate to compute its standard error?

    The p-value is computed assuming the null is true, and under the null there is a single common rate whose best estimate is the pooled rate.
  2. 2. A z statistic of 2.6 for a two-sided test corresponds to roughly which p-value?

    A value 2.6 standard errors from zero has two-sided tail probability near 0.009, below the usual 0.05 threshold.
  3. 3. In which situation is the normal approximation behind the z-test unreliable?

    Expected counts of conversions would be well below 5, so an exact method such as Fisher's exact test is safer.
  4. 4. A test with 2 million users per arm gives p = 0.001 for a lift of +0.02 percentage points. What is the best summary?

    Large samples make tiny effects detectable. Practical importance depends on the size of the effect, not the p-value.

Summary

  • The two-proportion z-test computes z = (p_B - p_A) / SE with a pooled SE and a normal-based p-value.
  • It requires independent users, enough expected conversions in each cell and a single, planned analysis.
  • Use pooled SE for the test and unpooled SE for the confidence interval.
  • Always weigh the effect size, not just the p-value.

Next: Confidence Intervals for Lift