Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 36 ยท UX in Teams and Metrics

A/B Testing for UX Decisions

Learn how to run A/B tests for design decisions: write a hypothesis, pick a metric, estimate sample size, read significance correctly and avoid common traps.

Usability tests tell you why people struggle, with a handful of participants. An A/B test tells you which of two designs performs better with real users at scale. You show version A (the control) to one random group and version B (the variant) to another, then compare a metric such as sign-up rate.

A/B testing answers "which is better?" but not "why?". Use it after you have good candidates from research and testing, not instead of them.

The structure of a good experiment

  1. Observation and hypothesis. "Users abandon the sign-up form at the phone number field. We believe making it optional will increase completed sign-ups."
  2. Primary metric. One metric decided in advance (sign-up completion rate). Add guardrail metrics that must not get worse (support tickets, refunds, load time).
  3. Variants. Change one meaningful thing so you can interpret the result; otherwise you learn that "something" helped.
  4. Random assignment. Each user is randomly and consistently placed in one group.
  5. Sample size and duration, decided before you start.
  6. Analysis using the plan you wrote at the beginning.

Reading the result

Say A converts at 10.0 percent and B at 11.5 percent. Is B better, or is the gap luck? A statistical test estimates how surprising the difference would be if there were no real effect. The p-value is the probability of seeing a difference at least this large by chance when A and B are actually the same. By convention p below 0.05 is called "statistically significant". It does not tell you the probability that B is better, nor how large or valuable the effect is; look at the confidence interval for the size of the lift.

Python Two-proportion z-test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 2.4-point lift is still inconclusive with 500 users per arm (the interval spans zero), while a smaller 1.5-point lift is clearly significant with 10,000. Sample size matters.

How many users do you need?

Before launching, estimate sample size from four inputs: your baseline rate, the minimum detectable effect (the smallest lift worth caring about), the significance level (usually 5 percent) and power (usually 80 percent: the chance of detecting a true effect of that size).

Python Sample size per variant

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Halving the effect you want to detect roughly quadruples the traffic you need. Small sites often cannot detect small effects, so prefer bigger, bolder changes or use qualitative methods.

Traps to avoid

Common A/B testing mistakes

Peeking Stopping when it first looks significant inflates false wins.
Too short Run full weeks to cover weekday habits.
Many metrics Test enough metrics and one will look good by luck.
Novelty effect Change itself attracts clicks at first.
1 / 2
  • Sample ratio mismatch: if a 50/50 split delivers 52/48 of users, assignment is broken; do not trust the result.
  • Segments: a result may differ for new versus returning users or mobile versus desktop; check, but treat post-hoc slices as hypotheses.
  • Local maxima: testing button colors endlessly will not fix a confusing flow.
  • Ethics: do not run tests that deceive or harm users, such as manipulative "dark patterns" (hidden costs, guilt-tripping buttons). A test can "win" while eroding trust.

When not to A/B test

Low traffic, one-time redesigns, or questions about why. Use usability testing, heuristic review or interviews instead. Also consider multivariate tests (several elements at once) only with large traffic.

Document and share

Record the hypothesis, setup, results (including inconclusive ones), decision and learning in a shared log. Inconclusive and negative results are valuable: they stop the team from repeating ideas.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. What does a p-value below 0.05 mean?

    A p-value measures surprise under no real difference; it says nothing about effect size.
  2. 2. Why decide sample size before starting?

    Peeking and stopping at the first significant result inflates false positives.
  3. 3. What is a guardrail metric?

    Guardrails such as page speed or refunds protect against wins that hurt elsewhere.

Summary

  • A/B tests compare variants on a pre-chosen metric with random assignment.
  • Interpret p-values with confidence intervals and effect size.
  • Plan sample size and duration upfront; avoid peeking and metric fishing.
  • Use experiments with, not instead of, qualitative research; avoid manipulative designs.

Next: UX in Agile Teams and Stakeholder Work