Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 37 ยท Getting Started

Key Terms and Metric Types

Learn the vocabulary of A/B testing (control, treatment, lift, null hypothesis, p-value, significance) and the main metric types, with runnable pandas examples.

A/B testing borrows words from statistics, product analytics and engineering, and teams use them loosely. Before the math starts, it helps to fix a shared vocabulary. This lesson defines the terms you will meet constantly and shows the three kinds of metrics you will analyze.

The vocabulary

Term Meaning
Variant / arm One version in the test. A/B has two arms; A/B/n has more.
Control The current experience (A), the baseline you compare against.
Treatment The new experience (B) you want to evaluate.
Randomization unit What gets assigned: user, session, device, account, page view.
Exposure The moment a unit actually experiences its variant.
Metric A number computed per unit (or per event) that you compare between arms.
Baseline rate The control group's value of the metric.
Effect / lift The treatment minus control difference. Absolute lift is B - A; relative lift is (B - A) / A.
Null hypothesis (H0) "The change has no effect." The default claim a test tries to reject.
Alternative (H1) "There is an effect" (two-sided) or "B is better" (one-sided).
p-value If H0 were true, how often would we see a difference at least this large from noise alone?
Significance level (alpha) The false-positive rate you accept, usually 0.05.
Confidence interval (CI) A range of effect sizes consistent with the data.
Power The chance of detecting an effect of a given size if it really exists.
MDE Minimum detectable effect: the smallest effect your test is designed to find.

Two of these cause the most confusion, so look at them closely.

Absolute vs relative lift. If control converts at 10% and treatment at 11%, the absolute lift is +1 percentage point and the relative lift is +10%. Always say which one you mean; "we increased conversion by 10%" is ambiguous.

The p-value is not the probability that B is better. It assumes the null is true and asks how surprising the data are. A p-value of 0.03 means: if the change did nothing, you would see a gap this big or bigger only 3% of the time. It does not say there is a 97% chance the change works.

Python Absolute and relative lift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three kinds of metrics

The kind of metric decides which statistical test fits. Almost everything you track falls into one of three shapes.

flowchart TD
    M["Metric per user"] --> B["Binary<br/>did it happen? 0 or 1"]
    M --> C["Continuous<br/>how much? revenue, time"]
    M --> K["Count<br/>how many? clicks, orders"]
    B --> BT["rate = mean of 0/1<br/>z-test, chi-square"]
    C --> CT["mean<br/>t-test, bootstrap"]
    K --> KT["mean per user<br/>t-test, Poisson ideas"]

Even a binary metric is a mean (the conversion rate is the average of zeros and ones), which is why the same toolkit covers all three.

  • Binary (proportion) metrics: each user either did the thing or did not. Conversion, click, signup, churn. The metric is the share who did.
  • Continuous metrics: each user has a number on a scale. Revenue per user, session length, page load time. Often skewed with a long tail of big spenders.
  • Count metrics: non-negative whole numbers. Items bought, pages viewed, searches. Usually also right-skewed.

Let us build a small table of users with one of each and compute the per-arm summaries with pandas.

Python One table, three metric types

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the revenue column. Most users spent nothing, and a few spent a lot. Skewed metrics like this need extra care, which t-tests for continuous metrics and later variance-reduction lessons address.

Unit of analysis vs unit of randomization

Two units matter and they should usually match. The unit of randomization is who is assigned (for example a user). The unit of analysis is what each row in your table represents (for example a page view). If you randomize by user but analyze by page view, rows from the same user are not independent, and your p-values come out too optimistic. The safe default: analyze at the same level you randomize, and compute metrics per randomization unit.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Control converts at 5% and treatment at 6%. What are the absolute and relative lifts?

    Absolute lift is 6% - 5% = 1 percentage point. Relative lift is 1 / 5 = 20%.
  2. 2. What does a p-value of 0.03 mean?

    The p-value is computed assuming the null is true. It measures how surprising the data are under "no effect," not the probability of either hypothesis.
  3. 3. Revenue per user is best described as which metric type?

    Revenue takes values on a numeric scale. It is usually right-skewed with many zeros and a long tail.
  4. 4. You randomize by user but compute your statistics treating each page view as an independent row. What is the risk?

    Repeated rows from one user are not independent. Treating them as independent understates the variance and overstates significance.

Summary

  • Know the vocabulary: control, treatment, unit, exposure, lift, null hypothesis, p-value, alpha, CI, power, MDE.
  • Say whether a lift is absolute or relative.
  • Metrics are binary, continuous or count, and each is a mean per unit.
  • Analyze at the level you randomize.

Next: Simulating Experiments in Python