Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 37 ยท Hypothesis Tests and Metrics

Confidence Intervals for Lift

Report A/B test results with confidence intervals. Compute absolute and relative lift intervals, understand coverage by simulation, and use the delta method.

A p-value answers a yes/no question ("is it different from zero?"). A confidence interval (CI) answers the question you actually care about: "how big is the effect, and how sure are we?" Good experiment reports lead with the interval, because it shows the size and the uncertainty at once.

What a confidence interval means

A 95% CI for the lift is built so that, if you repeated the experiment many times, about 95% of the intervals would contain the true lift. It is not a statement that "there is a 95% chance the true lift is in this interval" for one particular interval; it is a statement about the procedure's long-run reliability. In practice, read it as the range of effects consistent with the data.

The interval and the test agree: a 95% CI excludes zero exactly when the two-sided test at alpha = 0.05 is significant (when both use the same standard error).

Absolute lift interval

For the difference d = p_B - p_A, use the unpooled standard error:

SE = sqrt( p_A(1-p_A)/n_A + p_B(1-p_B)/n_B )

and the interval d +/- z * SE, with z = 1.96 for 95%.

Python 95% CI for the absolute lift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Higher confidence means a wider interval. The 95% interval excludes zero here, matching a significant z-test.

Relative lift interval

Stakeholders often prefer "+14% relative" to "+1.6 points." The relative lift is d / p_A, a ratio, so a simple add-and-subtract formula does not apply. Two practical approaches:

  1. Divide the absolute interval by the control rate. Quick but ignores the uncertainty in p_A.
  2. Delta method: approximates the standard error of the ratio p_B / p_A using calculus on the log scale: SE(log(p_B/p_A)) ~ sqrt( (1-p_A)/(n_A p_A) + (1-p_B)/(n_B p_B) ). Compute the interval for the log ratio and exponentiate.
Python Relative lift with the delta method

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two intervals are close here. The delta method is the better habit, especially when the control rate is small or uncertain.

Checking coverage by simulation

You can test the claim "95% of intervals contain the truth" by running thousands of experiments with a known lift and counting. This kind of coverage check is a very good way to trust a method.

Python Does a 95% CI really cover 95%?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Coverage lands near 95%, confirming the method. It also shows the typical width: that is how precisely this experiment can pin down the lift.

Reading intervals for decisions

flowchart LR
    A["CI entirely above 0<br/>and above the threshold"] --> S["Ship"]
    B["CI above 0 but<br/>reaches below the threshold"] --> I["Small or uncertain win:<br/>iterate or ship cheap changes"]
    C["CI straddles 0, narrow"] --> N["No meaningful effect:<br/>stop"]
    D["CI straddles 0, wide"] --> M["Inconclusive:<br/>need more data"]
    E["CI entirely below 0"] --> X["Harm: do not ship"]

Position of the interval relative to zero and to your practical threshold drives the decision.

Report intervals alongside the point estimate, for example: "Lift +1.6 points (95% CI +0.4 to +2.8)." That sentence carries the direction, the size and the uncertainty.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which standard error is used for the confidence interval of the difference in proportions?

    The interval does not assume the null (equal rates), so each arm's variance is estimated separately.
  2. 2. How does a 99% interval compare with a 95% interval for the same data?

    Greater confidence requires a larger multiplier on the standard error, so the interval is wider.
  3. 3. A 95% CI for the lift is (+0.2, +1.8) percentage points. What does that tell you about the two-sided test at alpha = 0.05?

    A 95% CI that excludes zero corresponds to a significant two-sided test at 0.05.
  4. 4. Why is the delta method used for relative lift?

    A ratio of two estimated rates does not have a simple additive standard error; the delta method approximates it.

Summary

  • A confidence interval shows the size and the uncertainty of the effect; lead your reports with it.
  • Absolute-lift CIs use the unpooled standard error; relative-lift CIs use the delta method on the log ratio.
  • A 95% CI is right about 95% of the time across repeated experiments, which you can verify by simulation.
  • Read the interval against zero and against your practical threshold.

Next: T-Tests for Continuous Metrics