Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 37 ยท Getting Started

The Experimentation Lifecycle

Walk through every stage of an A/B test, from idea and hypothesis to design, launch, analysis and decision, and write a pre-registration plan in Python.

Running an A/B test is more than splitting traffic. Good teams follow a repeatable process, because most failed experiments fail in the planning, not in the math. A test with a vague goal, the wrong metric or too few users produces a number that looks authoritative but means nothing.

This lesson gives you the map for the rest of the course. Each stage has its own lessons later; here you see how they fit together and what to write down before launch.

The seven stages

flowchart LR
    I["1. Idea<br/>and evidence"] --> H["2. Hypothesis"]
    H --> D["3. Design<br/>metrics, size, split"]
    D --> L["4. Launch<br/>and monitor"]
    L --> A["5. Analyze"]
    A --> X["6. Decide"]
    X --> LR["7. Learn<br/>and share"]
    LR -.-> I

The cycle repeats: what you learn from one test is the evidence for the next idea.

  1. Idea and evidence. Start from data or research: a funnel step with a big drop-off, support tickets, user interviews, an earlier test. "Let's try red" is not an idea; "users abandon at the shipping step because the total is a surprise" is.
  2. Hypothesis. State what you will change, which metric should move, in which direction, and why. See Writing Testable Hypotheses.
  3. Design. Choose the randomization unit, the primary metric, guardrail metrics, the minimum effect worth detecting, the sample size and the run time.
  4. Launch and monitor. Start with a small ramp, check that assignment is balanced and nothing is broken, then go to full traffic.
  5. Analyze. Once the planned sample is reached, run the pre-chosen test and report the effect with a confidence interval.
  6. Decide. Ship, iterate or stop, using a rule agreed before you saw the data.
  7. Learn and share. Record what you tried, what happened and what you now believe, including the losers.

Decide before you look

The most valuable habit in experimentation is to write the plan down before launch. When you choose the metric, the sample size and the decision rule after seeing results, it is very easy (and usually unintentional) to pick whichever analysis tells a nicer story. This is called p-hacking or the "garden of forking paths," and later chapters show how badly it inflates false positives.

A plan, sometimes called a pre-registration, can be as small as a structured record. Here is one in code so the fields are concrete:

Python An experiment plan as data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You do not need code to keep a plan, but a checklist you can run is a surprisingly effective way to stop sloppy tests before they start.

Where experiments go wrong

The stages are also a list of places to look when a result feels strange:

Stage Typical failure
Idea Testing trivia with no evidence it matters
Hypothesis No expected direction or reason, so any outcome "confirms" it
Design Too few users, wrong unit, metric that cannot move
Launch Bug in assignment; one group sees a broken page
Analyze Stopping when the p-value dips below 0.05, ignoring multiple metrics
Decide Shipping a statistically significant but tiny, costly change
Learn Winners are celebrated, losers are forgotten, so the team repeats them

A decision rule in code

The "Decide" stage is where analysis turns into action. A written rule removes arguments after the fact. A simple, common one compares the confidence interval of the lift with a threshold of practical importance.

Python A written decision rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that "not significant" is not one answer. An interval that is narrow around zero (the "too small" case) tells you to stop; a wide interval (the "noisy" case) tells you the test was too small to say anything. Telling those apart is a skill the next chapters build.

Roles on a real team

In practice an experiment involves several people: a product manager who owns the question, a designer or engineer who builds the variants, an analyst or data scientist who designs and reads the test, and often a platform team that provides assignment and logging. Small teams combine these roles, but the responsibilities stay the same: someone must own the hypothesis, someone must guard the validity of the data, and someone must make the call.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should the primary metric and decision rule be written down before launch?

    Choosing metrics and rules after seeing results lets bias creep in and inflates false positives. A written plan fixes the analysis in advance.
  2. 2. Which statement is the best hypothesis?

    A good hypothesis names the change, the metric, the direction and the reason, so the outcome teaches something either way.
  3. 3. A confidence interval for the lift is (-0.010, +0.030) and the smallest lift worth shipping is 0.01. What does the data say?

    The interval includes negative values and values above the practical threshold, so the experiment is too noisy to decide.
  4. 4. What is the last stage of the lifecycle, and why does it matter?

    Documenting outcomes builds shared knowledge and stops teams from repeating failed ideas.

Summary

  • An experiment moves through idea, hypothesis, design, launch, analysis, decision and learning.
  • Write the plan, metrics and decision rule before you see data.
  • "Not significant" can mean "no meaningful effect" or "not enough data"; the confidence interval tells you which.
  • Record every result, including the failures.

Next: Key Terms and Metric Types