Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 38 ยท Conversion Optimization

A/B Testing and Statistical Significance

Learn to run A/B tests correctly: hypotheses, sample size, z-test for conversion rates, p-values, confidence intervals and the mistakes that produce false winners.

An A/B test shows two versions of something (A, the control, and B, the variant) to randomly split groups of visitors and compares a metric. Because the split is random, a difference in results can be attributed to the change rather than to who happened to see it. It is the most reliable way to learn whether a change helps.

The catch is chance. Even identical versions will show slightly different numbers. Statistics tells you whether an observed difference is larger than random noise would plausibly produce.

Setting up a test

  1. Hypothesis: see conversion rate optimization.
  2. One primary metric: for example purchase conversion rate. Pick it before the test starts.
  3. Guardrail metrics: things that must not get worse (revenue per visitor, refund rate, page speed).
  4. Random assignment per user, kept consistent (a visitor who saw B keeps seeing B).
  5. Sample size and duration decided in advance.
  6. Run full weeks so that day-of-week patterns balance out.

Comparing two conversion rates

Suppose A converted 400 of 10,000 visitors (4.0 percent) and B converted 460 of 10,000 (4.6 percent). Is B really better?

The standard check is a two-proportion z-test:

  • Pooled rate p = (conversions A + conversions B) / (visitors A + visitors B)
  • Standard error = sqrt(p x (1 - p) x (1/nA + 1/nB))
  • z = (rateB - rateA) / standard error
  • The p-value is the probability of seeing a difference at least this large if there were truly no difference. A p-value below 0.05 is the common (but arbitrary) threshold for "statistically significant".
JS A/B test significance calculator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The confidence interval is more informative than the p-value alone: it shows the plausible range of the true improvement. If it runs from nearly zero to a large gain, the effect could be tiny.

Sample size: decide before you start

Small samples cannot detect small changes. The required sample depends on four things: your baseline rate, the smallest improvement worth detecting (the minimum detectable effect), the significance level (usually 5 percent) and the power (usually 80 percent, the chance of catching a real effect).

JS Sample size per variant

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice how detecting half the effect takes roughly four times the visitors. Low-traffic sites should test bold changes, not button shades.

Pitfalls that create false winners

  • Peeking: checking results daily and stopping when p dips below 0.05 inflates false positives dramatically. Either fix the sample size and look once, or use a sequential method designed for continuous monitoring.
  • Too many variants or metrics: with 20 metrics, one will look significant by luck. Choose one primary metric; treat the rest as exploratory (or correct for multiple comparisons).
  • Stopping at a lucky moment or ending the test after one or two days.
  • Sample ratio mismatch: if a 50/50 split delivers 52/48 on large traffic, the assignment is broken, so distrust the results.
  • Novelty effect: a new design attracts clicks simply because it is new; check whether the lift holds.
  • Ignoring segments or seasonality: a campaign launching during a test can distort it.
  • Confusing significance with importance: with huge traffic a 0.1 percent change can be significant but worthless. Look at the size of the effect.

A non-significant result does not prove there is no difference. It means the test could not detect one. Record it as "inconclusive", not "no effect".

Beyond a single test

  • A/A tests (two identical versions) check that your tool and process produce roughly 5 percent false alarms, not more.
  • Multivariate tests vary several elements at once and need much more traffic.
  • Bayesian methods report the probability that B is better, which many teams find easier to read, but they still require discipline about stopping rules.
  • Email subject-line tests and ad creative tests follow the same rules.

Keep a test log

Record the hypothesis, screenshots, dates, sample, result, interval and what you decided. Over time this log is one of the most valuable assets of a marketing team, because it stops you from repeating failed ideas.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a p-value below 0.05 mean in an A/B test?

    The p-value is the probability of data this extreme under no true difference.
  2. 2. Why is peeking at results daily and stopping early a problem?

    Repeated looks give many chances for chance to cross the threshold.
  3. 3. To detect a smaller improvement you need:

    Sample size grows quickly as the detectable effect shrinks.
  4. 4. A test is not significant. The best conclusion is:

    Absence of evidence is not evidence of absence.
  5. 5. A 50/50 split on 100,000 visitors delivers 56,000 to A and 44,000 to B. What should you suspect?

    A large deviation from the planned split signals a bug in randomization or tracking.

Summary

  • A/B tests use random assignment to separate cause from chance.
  • Choose one primary metric and the sample size before you start.
  • Use a z-test, p-value and confidence interval to judge a difference.
  • Avoid peeking, metric fishing and early stopping.
  • Treat non-significant results as inconclusive and keep a test log.

Next: Forms and Checkout Optimization