Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 37 ยท Experiment Design

Choosing Metrics - OEC and Guardrails

Choose the right A/B test metrics. Learn the overall evaluation criterion (OEC), primary and secondary metrics, guardrails, proxy metrics and how to avoid gaming.

The metric you choose decides what your experiment can learn. Pick one that is easy to move but meaningless, and you will ship changes that look like wins and help nobody. Pick one that is meaningful but slow and noisy, and you will never reach a conclusion. This lesson shows how to choose well.

The metric families

Four roles for metrics

Primary / OEC The one number that decides
Secondary Explain how the primary moved
Guardrail Must not get worse
Diagnostic Debug and explore

Only the primary metric carries the ship decision.

1 / 2
  • Primary metric (or OEC, overall evaluation criterion): the single metric, or single weighted combination, that answers your hypothesis. It should reflect long-term value you care about, such as purchase rate or weekly active users.
  • Secondary metrics: other success measures that help explain a result (add-to-cart rate when the primary is purchases).
  • Guardrail metrics: things you promise not to break: page load time, error rate, unsubscribe rate, refund rate, revenue. A win on the primary that trips a guardrail is not a win.
  • Diagnostic metrics: funnel steps and segments that help you understand why the result happened.

What makes a good primary metric?

Property Question to ask
Meaningful If this moves, does the business or the user really benefit?
Sensitive Can it move enough, within the test window, to be detected?
Attributable Can the change plausibly affect it directly?
Robust Is it hard to game and free of heavy outliers?
Measurable Do you log it correctly for every unit?

There is a tension between meaningful and sensitive. Long-term revenue is meaningful but takes months and is noisy. Clicks are sensitive but easy to inflate with clickbait. Proxy metrics bridge the gap: a short-term metric that has been shown (by past experiments or modeling) to predict the long-term one. Validate a proxy before trusting it.

Examples

Change Primary metric Guardrails
Shorter signup form Signup completion rate Spam signups, 7-day activation
Faster search ranking Searches with a click Page latency, zero-result rate
Discount banner Revenue per visitor Gross margin, refund rate
New onboarding tour Day-7 retention Support tickets, tour skip rate

Notice that "revenue per visitor" for a discount banner beats "conversion rate": a discount nearly always raises conversion, but it may lower revenue per visitor. The right primary metric accounts for the trade-off.

Metrics per unit and ratio metrics

Always define a metric per randomization unit so the rows are independent. "Revenue per user" is one number per user. A ratio like "click-through rate = clicks / impressions" mixes two random quantities, and computing it as total clicks over total impressions when users generate many impressions breaks independence. The safe version is to compute per-user and then average, or use the delta method (covered in Variance Reduction).

Python Per-user mean vs pooled ratio

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both numbers can be legitimate, but they answer different questions ("what share of impressions get a click" versus "what is the typical user's click rate"), and they have different variance. Choose deliberately and write it in the plan.

Guardrails in practice

For each guardrail, state the threshold in advance. Typical forms:

  • Non-inferiority: "page load time must not rise by more than 50 ms."
  • No-harm: "refund rate must not increase significantly."
Python Guardrail check against a threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Here the guardrail passes because even the upper bound is below the 50 ms limit. If the bound were higher, the right reading would be "we cannot rule out harm," which is different from "harm exists."

Gaming and Goodhart's law

"When a measure becomes a target, it stops being a good measure." A team rewarded for clicks will make buttons bigger and more tempting. Defend against this with guardrails that capture the cost (bounce rate, satisfaction, refunds) and by reviewing results qualitatively, not only numerically.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a guardrail metric?

    Guardrails (latency, errors, refunds) protect against wins that cost something important elsewhere.
  2. 2. Why can "conversion rate" be a poor primary metric for a discount banner?

    The metric should capture the trade-off you care about. Revenue per visitor or margin includes the cost of the discount.
  3. 3. What is a proxy metric?

    Proxies are sensitive, fast metrics that stand in for long-term value; they should be validated against the long-term metric.
  4. 4. A guardrail's 95% upper bound for added latency is 70 ms and the tolerance is 50 ms. What is the correct reading?

    The upper bound exceeds the tolerance, so a harmful slowdown is still plausible; collect more data or investigate.

Summary

  • Choose one primary metric (the OEC), supported by secondary, guardrail and diagnostic metrics.
  • A good metric is meaningful, sensitive, attributable, robust and measurable.
  • Define metrics per randomization unit; be explicit about pooled versus per-user ratios.
  • Set guardrail thresholds before launch, and beware of gaming.

Next: Randomization and Assignment Units