Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 37 ยท Sequential and Bayesian Tests

Group Sequential Tests and Alpha Spending

Learn how group sequential designs let you stop an A/B test early without inflating errors: Pocock and O'Brien-Fleming boundaries and Lan-DeMets alpha spending, with runnable code.

The peeking problem says that looking many times costs you error rate. A group sequential design accepts that cost openly: you plan a small number of interim looks (say 4 or 5) and give each one a stricter threshold than 1.96, so the total false positive rate across all looks stays at 5%.

The error rate is a budget. You have 5% to spend (this is called alpha) and you decide in advance how much to spend at each look. Spend little early, when the data is thin and noisy; spend the rest at the end.

Boundaries: the two classic shapes

At each look you compute the usual z-statistic and compare it with a boundary. If it crosses, you may stop.

  • Pocock uses the same boundary at every look (for 5 looks it is about 2.41 instead of 1.96). Easy to understand, but it makes the final analysis harder to pass, and it spends a lot of alpha early.
  • O'Brien-Fleming uses very strict early boundaries (above 4 at the first of five looks) that relax toward roughly 2.0 at the end. Early stopping is rare and needs a huge effect, but the final test is almost as sensitive as a fixed-horizon test. Most experimentation teams prefer this shape.

Two ways to spend alpha

Pocock Same bar at every look
O'Brien-Fleming Very strict early, relaxed late

Both keep total error at 5%

1 / 2

Alpha spending functions

Fixing the number of looks in advance is awkward in practice: the third look may happen on day 9 instead of day 7. Lan-DeMets alpha spending removes that problem. You pick a function that says how much cumulative alpha may be spent by information fraction t (the share of the planned sample collected so far, from 0 to 1). The looks can then happen whenever you like.

Two common functions:

  • O'Brien-Fleming-like: alpha(t) = 2 - 2 * Phi(z_{alpha/2} / sqrt(t))
  • Pocock-like: alpha(t) = alpha * ln(1 + (e - 1) * t)
Python Cumulative alpha spent at each fraction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the first rows: at 20% of the data the O'Brien-Fleming-like function has spent almost nothing, the Pocock-like one already about 30% of the budget. Both end at exactly 0.05 at t = 1.

Turning a spending plan into boundaries

At look k you want the probability of first crossing at that look to equal the alpha newly spent: alpha(t_k) - alpha(t_{k-1}). Exact values come from numerical integration (software such as the R package gsDesign does this). We can get close by simulating many no-effect experiments as random walks and picking each boundary from the paths that have not yet stopped.

Python Simulate boundaries for 5 equally spaced looks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The boundaries start high and fall toward a value near 2.0, and the overall false positive rate across all five looks comes out close to 0.05. (They are Monte Carlo estimates, so the first boundary is noisy; production tools compute them exactly.)

Using the plan during the experiment

Python Check a result against the boundary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that a z-statistic near 2.5 at look 2 (which would be "significant" in a fixed test) does not stop the test, because the bar there is 3.23. This is the price of looking.

Things the boundaries do not fix

  • Futility stopping. You may also want to stop when the effect is clearly absent. Futility bounds are separate and are usually non-binding (you may ignore them without hurting alpha).
  • Point estimates. A test stopped early at a crossing still overestimates the effect. Adjusted estimates and intervals exist and are part of proper software.
  • Pre-registration. The looks and the spending function must be chosen before launch. Choosing them after seeing data brings the peeking problem back.
  • Metric timing. Interim looks need metrics that are mature at each look; a 14-day retention metric cannot be evaluated on day 3.

If your team wants only one early-stop safety net, use an O'Brien-Fleming-like boundary with 3-5 looks plus a guardrail check for serious harm. It protects you from disasters at almost no cost in final power.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main idea of a group sequential test?

    Alpha is a budget split across the looks, so each look uses a stricter boundary than 1.96.
  2. 2. Which boundary shape is very strict early and close to 1.96 at the end?

    O'Brien-Fleming boundaries start high and relax, costing almost no power at the final analysis.
  3. 3. What does a Lan-DeMets spending function allow?

    The function depends on the information fraction, so looks can happen whenever you like. It must still be chosen in advance.
  4. 4. At look 2 of 5 your z is 2.5 and the O'Brien-Fleming-type boundary is 3.23. What do you do?

    2.5 is below 3.23, so the boundary is not crossed. Continue to the next planned look.

Summary

  • Group sequential tests give each interim look a stricter boundary so the overall false positive rate stays at 5%.
  • Pocock boundaries are flat; O'Brien-Fleming boundaries are strict early and near 1.96 at the end.
  • Lan-DeMets alpha spending lets looks happen at flexible times, using a pre-chosen spending function.
  • Boundaries can be estimated by simulation; real tools compute them exactly.
  • Stopped tests still overestimate effects, and the plan must be fixed before launch.

Next: Bayesian A/B Testing with the Beta-Binomial Model