Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 38 · Evaluation Metrics

Beyond Accuracy: Coverage, Diversity and Novelty

Measure what accuracy misses: catalog coverage, the Gini index, intra-list diversity, novelty and serendipity, plus MMR re-ranking to trade accuracy for variety.

A model can have excellent NDCG and still produce a boring product. If it recommends the same twenty blockbusters to everybody, it scores well on accuracy (those items are popular, so they are often hits) while the rest of the catalog is invisible and users learn nothing new. Beyond-accuracy metrics describe the shape of what the system recommends, not only whether each item was a hit.

These metrics matter commercially (long-tail items, supplier fairness, user retention) and are tracked next to accuracy in every serious evaluation.

The main metrics

  • Catalog coverage: the share of all items that appear in at least one user's top-N list. Low coverage means most of the catalog is never shown.
  • Gini index of exposure: how unevenly the recommendation slots are spread over items. 0 means every item is shown equally often; values near 1 mean a few items take nearly all slots.
  • Intra-list diversity (ILD): the average pairwise dissimilarity of the items within one list. A list of ten sequels has low ILD; a mix of genres has high ILD. You need an item similarity (for example cosine on genre vectors) to define it.
  • Novelty: how unfamiliar the items are. A common definition is the mean self-information of the recommended items, −log2(popularity share): rare items give high values.
  • Serendipity: relevant and surprising, for example a hit that a simple popularity or content baseline would not have suggested. It is hard to measure offline and often evaluated with surveys.
  • Calibration: whether the mix of genres in a list matches the mix in the user's history (a user who watches 70% drama and 30% comedy should not get 100% drama).

Questions accuracy cannot answer

Coverage How much of the catalog is ever shown?
Diversity Are the items in one list different?
Novelty Is it more than the obvious hits?
Calibration Does the list match the user's taste mix?

Track them next to NDCG, never instead of it.

1 / 2

Computing them

The next block builds item genres (each item has a main genre and the users' taste groups prefer them), produces top-10 lists with a popularity model and with item-item collaborative filtering, and reports coverage, Gini, ILD and novelty for each.

Python Coverage, Gini, diversity and novelty

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the lines. Popularity shows the textbook pattern: the same few items are shown to everyone, so coverage is small and the exposure Gini is high. The personalized model spreads the slots over more of the catalog. Note that the list diversity inside one list (ILD) can behave differently from the coverage across users, since collaborative filtering tends to fill a list with items from one taste group, which is exactly what we tackle next.

Re-ranking for diversity with MMR

A straightforward way to raise list diversity is a re-ranking step after scoring. Maximal marginal relevance (MMR) builds the list greedily: at each step it picks the item that maximizes

TEXT
λ · relevance(item)  −  (1 − λ) · max similarity(item, items already chosen)

With λ = 1 you get the original ranking; smaller λ pushes the list toward variety at some cost in accuracy. The block shows both sides of that trade-off by measuring the ILD and a simple relevance proxy (recall of held-out plays) for several λ.

Python MMR: trading accuracy for diversity

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Going from λ = 1 to 0.7, ILD rises from 0.34 to 0.47 while recall falls from 0.40 to 0.36: that is the accuracy-diversity trade-off in numbers. The 0.7 and 0.4 rows coincide because this toy catalog has only three genres, so the penalty already saturates. The right point depends on the product. A shopping home page often wants variety; a "because you watched X" row wants similarity. Choose λ with an online test rather than by looking at offline metrics alone.

Practical notes

  • Always report a bundle. For example NDCG@10 with coverage@10 and ILD@10, so a gain in accuracy that destroys variety is visible.
  • Popularity bias. Models trained on logs over-recommend popular items. Metrics like novelty and the Gini index expose it; re-weighting training examples or re-ranking can reduce it.
  • Fairness for providers. Coverage and exposure Gini can be computed per creator or seller to check that small providers receive a fair share of the slots.
  • Offline limits. Novelty and diversity can be computed offline, but whether users like the change needs an online experiment (next lesson).

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. What does catalog coverage measure?

    Low coverage means most of the catalog is never recommended to anyone.
  2. 2. A Gini index of exposure close to 1 means

    The Gini index measures inequality. Near 1 means exposure is concentrated on very few items.
  3. 3. In MMR, what does lowering λ do?

    A smaller λ weighs the similarity penalty more, so selected items are more different from each other.
  4. 4. How is novelty often computed?

    Rare items have low popularity share and therefore high self-information.

Summary

  • Accuracy metrics do not describe what a system shows: add coverage, Gini, diversity, novelty and calibration.
  • ILD needs an item similarity; novelty uses item popularity; coverage and Gini describe exposure across the catalog.
  • MMR re-ranking trades relevance for variety through one parameter.
  • Report accuracy and beyond-accuracy metrics together, then validate with online tests.

Next: Offline Evaluation and A/B Testing