Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 38 ยท Evaluation Metrics

Precision and Recall at K

Evaluate top-N recommendation lists with precision@K, recall@K, F1@K and hit rate: definitions, worked numbers, averaging over users, and common evaluation mistakes.

A recommender shows a short list, so the natural questions are: how many of the items in that list were relevant, and how many of the relevant items did the list find? These are precision at K and recall at K. They treat the list as a set: all that matters is whether each item is a hit, not where in the list it sits (that is the job of ranking metrics).

You have already used recall at 10 in the previous lessons to compare models. Here you will define it properly and learn the variants and the traps.

Definitions

For one user, let Rec@K be the top K recommended items (excluding items the user already had in the training data) and Rel the set of items that are relevant in the held-out data, for example the items they used in the test period.

TEXT
Precision@K = |Rec@K โˆฉ Rel| / K
Recall@K    = |Rec@K โˆฉ Rel| / |Rel|
F1@K        = 2 ยท P ยท R / (P + R)
HitRate@K   = 1 if |Rec@K โˆฉ Rel| โ‰ฅ 1, else 0
  • Precision@K answers "how much of what I showed was good?". It is bounded by |Rel| / K: a user with two relevant items can never reach precision 1.0 at K = 10.
  • Recall@K answers "how much of what they wanted did I find?". It rewards a longer list K.
  • Hit rate@K (also called coverage of users or "success at K") is the share of users for whom at least one relevant item appears. With a single held-out item per user (leave-one-out), it equals recall.

Report each metric as an average over users, and decide beforehand whether users with an empty Rel are skipped (usual) or counted as zero.

Precision vs recall at K

Precision@K Hits divided by K
Recall@K Hits divided by relevant items
Hit rate@K Any hit at all?

Raising K tends to raise recall and lower precision.

1 / 2

A worked example

A user's held-out relevant items are {4, 9, 15}. The model recommends [9, 2, 4, 7, 12] at K = 5. Hits are 9 and 4, so precision@5 is 2/5 = 0.4 and recall@5 is 2/3 โ‰ˆ 0.667. The code computes these and sweeps K.

Python Precision, recall and F1 at K

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As K grows, recall never decreases (more slots can only find more), while precision tends to fall because every extra slot is a new chance to include a miss. F1 balances the two, but in recommendation the right K is usually fixed by the interface (a carousel with 10 cards), so evaluate at that K.

Averaging over users and comparing models

The next block builds a small implicit-feedback dataset with taste groups, holds out some interactions and scores three systems at K = 10: random, popularity and a simple item-item collaborative model. It shows how the model ordering holds for precision, recall and hit rate, and how the metrics are averaged over users.

Python Comparing three recommenders at K = 10

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A few things to read from the table. Precision values look low in absolute terms because each user has only a handful of held-out items and K = 10 slots: that cap is why you should compare models on the same split rather than chase an absolute number. A collaborative model should clearly beat random, and it should beat popularity too because popularity gives everyone the same list.

Common mistakes

  • Counting seen items as hits or recommendations. Remove the user's training items from the candidates, or recall will be inflated by items the model already knew.
  • Treating unobserved as irrelevant without saying so. An unobserved item may be a great recommendation the user never saw. Offline precision is therefore a pessimistic estimate; online tests (A/B testing) measure the real effect.
  • Mixing different candidate sets. Two models must rank the same candidate pool.
  • Reporting only the mean. Also look at the per-segment values (new vs heavy users, popular vs tail items).
  • Leaking the future. Use the time-based split; shuffling puts test-period behavior into training.
  • Ignoring K. Always state K. Precision@5 and precision@50 are different metrics.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A user has 3 relevant items. The top-5 list contains 2 of them. What are precision@5 and recall@5?

    Precision is hits/K = 2/5 = 0.4. Recall is hits/|Rel| = 2/3 = 0.667.
  2. 2. What usually happens when K increases?

    More slots can only find more relevant items, but each extra slot is another chance to be wrong.
  3. 3. Why must a user's training items be excluded from the ranked list before computing recall@K?

    Known items are not recommendations. They distort the list and the metric.
  4. 4. Why can offline precision understate the quality of a recommender?

    Unobserved does not mean disliked. Only an online experiment sees the effect of showing new items.

Summary

  • Precision@K is hits divided by K; recall@K is hits divided by the number of relevant items.
  • Hit rate@K asks whether any relevant item was found; F1 balances precision and recall.
  • Average over users, exclude already-seen items and fix K to the interface size.
  • Compare models on identical splits and candidate pools, using time-based splits.
  • Offline precision is a pessimistic estimate, since unseen items count as misses.

Next: Ranking Metrics: MAP, MRR and NDCG