Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 37 ยท Experiment Design

Writing Testable Hypotheses

Learn to write A/B test hypotheses that name the change, metric, direction and reason, and understand null and alternative hypotheses and one-sided vs two-sided tests.

A hypothesis is a prediction you can be wrong about. In an experiment it links a change you control to a result you can measure, and it carries a reason. Without that reason, a test can only tell you that something happened, never what to do next.

Many teams skip this step and write goals like "improve the homepage". That cannot be falsified: any result can be spun as progress. A good hypothesis commits you in advance, which is exactly what makes the result informative.

The anatomy of a good hypothesis

A useful template has four parts:

If we change X, then metric Y will increase/decrease, because reason Z.

  • Change (X): the concrete difference between control and treatment.
  • Metric (Y): the primary metric, named before launch.
  • Direction: up or down, and ideally by how much.
  • Reason (Z): the user behavior or mechanism you believe in.

Compare a weak and a strong version:

Weak Strong
"Test a new homepage" "If we replace the carousel with one static hero and a single CTA, then click-through to signup will rise by at least 5% relative, because users currently ignore auto-rotating slides."
"Make checkout faster" "If we remove the account-creation step for first-time buyers, then purchase completion will rise, because 30% of abandoners leave at that step."

The reason matters most after the test. If the hypothesis fails, the reason tells you which belief was wrong. If it succeeds, the reason tells you where else the idea might apply.

Null and alternative hypotheses

The statistical hypotheses are a formal restatement of your product hypothesis.

  • Null hypothesis, H0: the change has no effect. For a conversion rate, p_B = p_A.
  • Alternative hypothesis, H1: the change has an effect. Two-sided: p_B != p_A. One-sided: p_B > p_A.

A test never proves H1. It asks whether the data are surprising enough under H0 to reject it. Failing to reject H0 does not prove "no effect"; it may only mean the test was too small to see one.

flowchart LR
    P["Product hypothesis<br/>'big hero raises signups'"] --> S["Statistical hypotheses<br/>H0: p_B = p_A<br/>H1: p_B != p_A"]
    S --> T["Test statistic<br/>and p-value"]
    T --> R{"p < alpha?"}
    R -- yes --> Rej["Reject H0<br/>evidence of an effect"]
    R -- no --> Fail["Fail to reject H0<br/>no evidence yet"]

"Fail to reject" is not "accept". Absence of evidence is not evidence of absence.

One-sided or two-sided?

A two-sided test looks for a difference in either direction. A one-sided test only looks for an improvement. One-sided tests need fewer users to detect a win, because all the 5% of tail probability sits on one side. The catch: they cannot flag a harmful change, and it is tempting to pick the side after seeing the data, which is cheating.

For product experiments the standard advice is: use two-sided tests by default. A change that makes things worse is information you need. Use one-sided only when a harmful result would lead to the same decision as no result, and decide that before launch.

Python One-sided vs two-sided p-values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same data are "significant" one-sided but not two-sided. That is exactly why the choice must be made in advance. If you flip to one-sided after a two-sided result lands at p = 0.07, you have quietly raised your false-positive rate.

Hypotheses should be falsifiable and specific

Check each hypothesis against four questions:

  1. Could the data prove it wrong? "Users will like it" is not falsifiable; "7-day retention rises" is.
  2. Is the metric measurable in the test window? Lifetime value is not; first-week purchase rate may be.
  3. Is the expected size stated? Without a minimum effect you cannot compute a sample size (Power and Sample Size).
  4. Is the mechanism plausible? Surprising results need extra evidence; a hypothesis with no mechanism invites fishing.

A simple hypothesis checker

The following tool is deliberately crude (it only looks at wording), but it shows how mechanical the "is this hypothesis complete?" check can be, and it is a handy team habit.

Python Does the hypothesis have all four parts?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Primary, secondary and exploratory

Most experiments move many numbers. Decide up front which one answers the hypothesis (the primary metric, exactly one), which support it (secondary), which protect you (guardrails, next lesson), and which are just for exploring. Only the primary metric carries the decision; secondary and exploratory results are leads for future tests, not proof.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which part of a hypothesis tells you what to do after the test, win or lose?

    The reason states the belief being tested. Whether the result confirms or contradicts it, you learn which assumption held.
  2. 2. What does failing to reject the null hypothesis mean?

    Failing to reject is absence of evidence, not proof of no effect. A small or noisy test may simply be unable to see one.
  3. 3. Why is it risky to choose a one-sided test after seeing the data?

    Picking the tail after seeing the data halves the p-value in your favor, inflating the false-positive rate. The side must be chosen in advance.
  4. 4. How many primary metrics should an experiment have?

    A single primary metric keeps the decision clean and avoids multiple-comparison problems.

Summary

  • Write hypotheses as "if change, then metric moves, because reason".
  • H0 says no effect; a test rejects it or fails to, and failing to reject is not acceptance.
  • Prefer two-sided tests; choose any one-sided test before launch.
  • Name exactly one primary metric, plus secondary and guardrail metrics.

Next: Choosing Metrics: OEC and Guardrails