Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 37 ยท Getting Started

What Is A/B Testing?

Learn what an A/B test is, why randomized experiments beat guesses and before-and-after comparisons, and run your first simulated experiment in Python.

An A/B test is a controlled experiment. You show version A of something (a button, a price, an email, a ranking algorithm) to one random group of people and version B to another, then compare a number you care about, such as how many people sign up. Because the groups were formed by chance, any lasting difference between them can be traced back to the change you made.

That single idea powers product decisions at companies of every size. This course teaches it end to end: how to design a test, how to analyze it, how many people you need, how to peek at results safely, how to avoid the classic traps, and how experimentation platforms work. Every example runs in your browser with Python, using simulated data so the "true" answer is known and you can check your own conclusions.

What runs here. All code in this course runs for real in your browser (Python, NumPy, pandas, SciPy and matplotlib in Pyodide). The data is simulated, not collected from real users, so we can see how a method behaves when we know the truth. Real platforms such as Optimizely, Statsig or an in-house system are described in words, not run.

The core idea: compare randomized groups

Suppose your checkout page has a blue "Buy" button and you wonder whether a green one sells more. You split visitors at random. Half see blue (the control, A), half see green (the treatment, B). After a week you count how many in each group bought.

The shape of every A/B test

Randomize Each user gets A or B by chance
Expose Each group sees only its version
Measure Same metric for both groups
Compare Is the gap bigger than luck?

Randomization is what makes the comparison fair.

1 / 2

The word "random" is doing the heavy lifting. If you let the most eager customers pick green, or you only ship green on Mondays, the groups differ in more than the button, and you can no longer say the button caused the difference.

Why not just compare before and after?

A common shortcut is to ship the new version to everyone and compare this week with last week. That mixes the effect of your change with everything else that changed: a holiday, a marketing campaign, a competitor outage, the weather. A concurrent control group experiences all of those too, so they cancel out.

flowchart LR
    U["All users"] --> R{"Random<br/>split"}
    R --> A["Group A<br/>old version"]
    R --> B["Group B<br/>new version"]
    A --> MA["metric A"]
    B --> MB["metric B"]
    MA --> D["difference<br/>= effect + noise"]
    MB --> D

Both groups live through the same week, so the only systematic difference is the change you tested.

Your first simulated experiment

We will invent a world where the old page converts 10% of visitors and the new page converts 11%. In real life you never know these numbers; here we do, which lets us see how well a test recovers the truth.

Python A first A/B test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The observed gap will not be exactly one percentage point. Each group is a random sample, so its rate wobbles around the truth. The central question of A/B testing is: is the gap we see bigger than the wobble we would expect from chance alone? Later lessons answer that with hypothesis tests and confidence intervals.

Noise is real: run it again

To feel the wobble, repeat the same experiment many times with different random draws and look at how much the observed lift varies. In every repetition the true lift is exactly +1 point.

Python The same test, 1,000 times

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The average lands near +1 point, so the experiment is unbiased: it is right on average. But single runs scatter, and in some of them the better version looks worse. That is why you never trust a single raw difference without measuring its uncertainty.

What A/B testing is good for

  • Choosing between concrete alternatives: headline wording, page layouts, onboarding flows, pricing pages, recommendation algorithms.
  • Measuring the size of an effect, not just its direction, so you can weigh it against its cost.
  • Catching harm: a "harmless" change that quietly slows the page or breaks checkout on one browser shows up in the guardrail metrics.
  • Settling arguments with evidence instead of seniority.

What it is not good for

  • Tiny audiences: with a few hundred visitors a month, most tests cannot detect anything realistic (see Statistical Power).
  • Long-term or network effects: if users influence each other (marketplaces, social feeds), simple randomization can mislead.
  • Questions of "why": a test says whether B beat A, not why. Pair it with user research.
  • One-off, irreversible decisions such as a rebrand where you cannot show two versions.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is randomly assigning users to A or B important?

    Random assignment balances known and unknown differences between users, so the only systematic difference between the groups is the change you made.
  2. 2. What is the main flaw of comparing this week with last week after shipping a change to everyone?

    Seasonality, campaigns and outages affect the before and after periods differently. A concurrent control group experiences the same conditions as the treatment group.
  3. 3. In the 1,000-repetition simulation, why did B sometimes look worse than A even though it was truly better?

    Sampling noise makes single experiments scatter around the true effect. Statistical tests quantify that noise.
  4. 4. Which question is an A/B test poorly suited to answer?

    A test measures behavior differences, not motives. Interviews and surveys help explain why.

Summary

  • An A/B test randomly splits users into control (A) and treatment (B) and compares one metric.
  • Randomization and a concurrent control make the comparison fair; before-and-after comparisons are confounded by time.
  • Observed differences contain noise, so every result needs a measure of uncertainty.
  • Tests are strong for choosing between concrete options and weak for tiny audiences, network effects and "why" questions.

Next: The Experimentation Lifecycle