Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 38 ยท Evaluation Metrics

Offline Evaluation and A/B Testing

Avoid misleading offline results: random vs time splits, leakage, selection bias, then confirm with online A/B tests: metrics, sample size, significance and guardrails.

Offline metrics are fast and cheap: you replay logged behavior and score a model against it. But the logs were produced by another recommender and by users who only saw what that system chose to show. So an offline number is a hint, not the truth. Online experiments (A/B tests) measure what really matters, but they cost time and risk real users. A good workflow uses offline evaluation to throw out bad ideas and A/B tests to confirm the good ones.

Offline pitfalls

1. Random splits leak the future. Randomly shuffling interactions into train and test lets the model see behavior from after the test events. It inflates scores for anything that tracks trends. Use a time-based split (train on the past, test on the future) as shown in the dataset lesson. For per-user evaluation, hold out each user's last interactions.

2. Data leakage in features. Item statistics (popularity, average rating) must be computed on training data only. If the test-period clicks went into a "popularity" feature, you evaluate on information the model could not have had.

3. Selection bias (exposure bias). Users only interact with items they were shown. A model that happens to resemble the old production recommender wins offline simply because the logs came from that recommender. A held-out item that the new model recommends but the users never saw counts as a miss even if they would have loved it.

4. Popularity bias in the metric. Hit-based metrics reward recommending popular items, because popular items are likely to be in the test set. Compare with a popularity baseline and also look at long-tail performance.

5. Evaluating on the full catalog vs sampled negatives. Some papers rank the test item against 100 random negatives instead of all items. The results are not comparable with full-ranking results, and sampled metrics can even reverse the order of two models. Prefer ranking against the full candidate set.

6. Tuning on the test set. Repeated peeking tunes the model to the test split. Keep validation separate and look at test once.

Seeing the leak

The block simulates items whose popularity rises and falls over time. A "most popular" model is trained either on a random 80% of events or on the first 80% of the timeline, and then scored on the held-out events by hit rate at 5 (the fraction of test events whose item is in the model's top 5).

Python Random split vs time split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same model scores about 0.13 on the random split, because the training set contains events from the test period, and almost 0 on the time split. The time split gives the number you would actually have seen in production: in this extreme toy world the items that were popular early are no longer the ones in demand at the end. Whenever behavior drifts (almost always), a random split is optimistic.

Online experiments: A/B tests

In an A/B test, you randomly assign users to the control (the current system) or the treatment (the new model), run both at the same time and compare a business metric: click-through rate (CTR), conversion, watch time, retention, revenue.

Checklist for a sound test:

  1. Pick one primary metric and the minimum improvement you care about before starting. Add guardrail metrics that must not get worse (latency, complaints, returns, unsubscribes).
  2. Randomize by user, not by request, so a person has a consistent experience and the samples are independent.
  3. Compute the sample size in advance from the baseline rate, the effect you want to detect and the power.
  4. Run whole weekly cycles (at least one, often two) to cover day-of-week effects. Do not stop early the moment the result looks significant ("peeking" inflates false positives).
  5. Check for sample ratio mismatch: if a 50/50 split produces 52/48, something is broken in assignment.
  6. Beware network effects: if treated users change what control users see (shared inventory, social feeds), simple user-level randomization is biased.

The statistics of a conversion test

For a rate such as CTR, compare two proportions with a z-test. With pฬ‚ = x/n in each arm and the pooled rate p, the statistic is z = (pฬ‚_B โˆ’ pฬ‚_A) / sqrt(p (1โˆ’p) (1/n_A + 1/n_B)). Roughly, |z| > 1.96 means significant at the 5% level (two-sided). The required sample size per arm for a relative lift is about 16 ยท p(1โˆ’p) / ฮดยฒ, where ฮด is the absolute difference you want to detect (a rule of thumb for 80% power at 5%).

Python Two-proportion z-test and sample size

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first test shows a 0.55 percentage point lift. Whether it counts as significant depends on both the lift and the sample size: compare the p-value and confidence interval of the first and the last test, and note that the identical relative lift on a tenth of the users is far from convincing. The sample-size line shows why small effects need tens of thousands of users per arm.

Other online methods

  • Interleaving: show a single list that mixes items from the two models and see which model's items get clicked. It needs far fewer users than a classic A/B test for ranking comparisons, but it only compares rankers, not whole-page effects.
  • Multi-armed bandits: shift traffic toward the better variant while the test runs, useful when exploring many variants (see Exploration, Feedback Loops and Fairness).
  • Counterfactual (off-policy) evaluation: estimate how a new policy would have performed from logs using inverse propensity weighting. It needs logged propensities (the probability the old system showed each item).
  • Holdouts: keep a small permanent control group to measure the long-term value of the whole recommender.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a random train/test split risky for recommender evaluation?

    A time-based split reproduces the real situation of predicting the future from the past.
  2. 2. What is exposure (selection) bias in logged data?

    Interactions exist only for shown items, so models that mimic the old recommender look better offline.
  3. 3. Which is a good A/B testing practice?

    Pre-registering the metric and the sample size avoids peeking and cherry-picking. Randomize by user.
  4. 4. What is a guardrail metric?

    Guardrails protect against wins on the primary metric that hurt something important.

Summary

  • Offline evaluation is fast but biased by time leakage, exposure bias, popularity and tuning on test data.
  • Use time-based splits, compute features on training data only and compare against a popularity baseline.
  • A/B tests measure real impact: one primary metric, guardrails, user-level randomization and a pre-computed sample size.
  • Use z-tests with confidence intervals, avoid peeking and check for sample ratio mismatch.
  • Interleaving, bandits and off-policy estimates are useful complements.

Next: The Cold Start Problem