Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Variance Reduction

The Delta Method for Ratio Metrics

Analyze click-through rate and revenue-per-session correctly when you randomize by user: why the naive test fails, the delta method standard error and a bootstrap check in Python.

Many of the metrics people care about are ratios: click-through rate (clicks / page views), revenue per session, items per order, average watch time per video. They look like simple averages but are hiding a trap. You randomize by user, while the numerator and denominator are summed over many events per user. Events from the same user are not independent, and the standard formula for a proportion pretends they are.

The delta method gives a correct variance for a ratio of two user-level totals. It is what experimentation platforms use under the hood for CTR and similar metrics.

The trap: treating events as independent

Suppose user i has x_i page views and y_i clicks. The metric is R = sum(y) / sum(x). The naive test treats every page view as a separate Bernoulli trial and uses sqrt(R (1 - R) / total_views). But a user who loves clicking clicks on many of their views, and heavy users contribute many views. Correlated events mean less information than the naive formula assumes, so the naive standard error is too small and the false positive rate too high.

Ratio metrics: unit mismatch

Randomize by user Users are independent
Metric per event Clicks / views
Events cluster Same user, correlated
Delta method Variance at user level

Analyze at the randomization unit

1 / 2

The formula

Per group, with n users, user totals x_i (denominator) and y_i (numerator), and R = sum(y) / sum(x):

TEXT
resid_i = y_i - R * x_i
Var(R)  = Var(resid) / (n * mean(x)^2)

In words: treat each user's "excess" over what the ratio predicts as the unit of analysis, then scale by the average denominator. The difference between groups has variance Var(R_T) + Var(R_C), and you test it with a z-statistic.

Python Delta method versus the naive standard error

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The delta-method standard error is about 20% larger than the naive one (the exact ratio depends on how much users differ). That is the honest uncertainty: the naive formula would make many A/A tests look significant.

Check it with a bootstrap

A bootstrap resamples users (not events) with replacement and recomputes the ratio. It makes no formula assumptions, so it is a good sanity check of the delta method.

Python Bootstrap the CTR at user level

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two agree closely, confirming that the delta method describes the user-level uncertainty well, at a fraction of the compute cost.

False positive rates in A/A tests

The real test of a method is how often it cries wolf when there is no effect.

Python A/A test: naive versus delta method

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The naive test rejects nearly twice as often as the nominal 5% in A/A experiments. The delta-method test stays near the nominal rate.

Where else do ratios appear?

Revenue per session, orders per user with a per-order metric, average rating, bounce rate, error rate per request. Whenever the numerator and denominator are both summed over events within a user, use the delta method (or a bootstrap of users), or change the metric so that the denominator is one per user.

If you can redefine the metric as a per-user average (clicks per user instead of clicks per view), a plain t-test is already valid. Ratios of totals answer a different question: they weight heavy users more. Choose deliberately, as described in choosing metrics.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is the naive standard error for CTR too small when you randomize by user?

    Clustering within users means less independent information than the number of events suggests.
  2. 2. In the delta method for R = sum(y) / sum(x), what is analyzed per user?

    The residual y_i - R x_i linearizes the ratio, and its variance drives the standard error.
  3. 3. What does a user-level bootstrap resample?

    Resampling the randomization unit preserves within-user correlation.
  4. 4. You change the metric to clicks per user. Which test is now valid?

    With one value per user, observations are independent and the usual t-test applies.

Summary

  • Ratio metrics sum numerator and denominator over events, but randomization happens at the user level.
  • The naive event-level formula understates the uncertainty and inflates false positives.
  • The delta method uses Var(R) = Var(y - R x) / (n * mean(x)^2) per group.
  • A user-level bootstrap is a good check; per-user averages avoid the issue entirely.

Next: Novelty, Primacy and Interference