Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 38 ยท Evaluation Metrics

Ranking Metrics: MAP, MRR and NDCG

Reward relevant items that appear early in the list: average precision, MAP, reciprocal rank, MRR, DCG and NDCG with graded relevance, worked examples and runnable code.

Precision and recall at K treat the list as a bag: finding the relevant item in slot 1 or slot 10 scores the same. Users do not behave that way. They look at the first few cards, so a good ranking puts relevant items early. Ranking metrics add a position discount. The three you will meet constantly are MRR, MAP and NDCG.

Reciprocal rank and MRR

The reciprocal rank (RR) for one user is 1 / position of the first relevant item in the list (0 if there is none). The mean over users is MRR. It answers "how far down must I go to find the first thing that works?" and suits tasks with one right answer, such as next-item prediction or search.

TEXT
RR = 1 / rank_of_first_hit        MRR = mean of RR over users

If the first hit is at position 1, RR is 1.0; at position 2, 0.5; at position 10, 0.1.

Average precision and MAP

Average precision at K (AP@K) looks at every relevant item found. Walk down the list; whenever you hit a relevant item at position p, record the precision at that point, precision@p. Average those recorded values (dividing by the number of relevant items, capped by K). MAP@K is the mean of AP over users.

TEXT
AP@K = (1 / min(|Rel|, K)) ยท ฮฃ over hit positions p โ‰ค K of precision@p

AP is high only when the relevant items are packed toward the top: a hit in slot 1 gives a precision of 1.0, a hit in slot 5 only at best 0.2 for that term.

DCG and NDCG with graded relevance

Sometimes relevance has grades: a purchase is worth more than a click, a 5-star rating more than a 4-star. Discounted cumulative gain (DCG) sums each item's gain divided by a logarithmic position discount:

TEXT
DCG@K  = ฮฃ over positions i = 1..K of  gain_i / log2(i + 1)
NDCG@K = DCG@K / IDCG@K

IDCG@K is the DCG of the best possible ordering of that user's relevant items, so NDCG always lies between 0 and 1, which lets you average across users with different numbers of relevant items. The log discount means position 1 has weight 1, position 2 about 0.63, position 3 0.5, position 10 about 0.29. With binary relevance, the gain is 1 for a hit and 0 otherwise. Some papers use the gain 2^rel โˆ’ 1, which stresses highly relevant items more; state which form you use.

Which ranking metric?

MRR Where is the first hit?
MAP All hits, early ones count more
NDCG Graded relevance, normalized 0 to 1

All three reward early relevant items.

1 / 2

Computing them

The functions below take a ranked list of item ids and the set (or graded dictionary) of relevant items. Two rankings contain the same two relevant items, one with the hits at the top and one with them near the bottom.

Python RR, AP and NDCG for two rankings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both rankings find both relevant items, so precision@5 and recall@5 are identical for them (0.4 and 1.0), yet RR, AP and NDCG separate them sharply. The last line shows that NDCG with graded gains also cares about the order of the two relevant items: putting the better item (a, grade 3) first is worth more than the reverse.

Comparing systems on a dataset

Now the same metrics averaged over users. The example uses an interaction dataset with taste groups, holds out some plays, and compares a popularity ranking with an item-item collaborative model using MRR, MAP@10 and NDCG@10 with binary gains.

Python MRR, MAP@10 and NDCG@10 on a dataset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three metrics tell the same story here, and that is typical when one model is clearly better. They can disagree when models differ in where they place hits, which is exactly why you should look at more than one.

Choosing and reporting

Situation Good primary metric
One correct next item (sessions, leave-one-out) MRR, hit rate@K
Several relevant items, binary labels MAP@K or NDCG@K
Graded feedback (purchase > click, star ratings) NDCG@K
Carousel of fixed size Metrics at that K

Always report the cutoff (@10), say how you handle users with no relevant items (usually skip them), and give variability (a confidence interval from bootstrapping users is a good habit). Two models with NDCG@10 of 0.231 and 0.233 are probably the same.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A user's first relevant item is at position 4 in the list. What is the reciprocal rank?

    RR = 1 / rank of the first hit = 1/4 = 0.25.
  2. 2. Why is NDCG normalized by IDCG?

    IDCG is the DCG of the best possible ordering, so NDCG = 1 means a perfect ranking for that user.
  3. 3. Two lists have the same relevant items, one with hits at positions 1 and 2, the other at 4 and 5. Which metrics tell them apart?

    Precision and recall at K are position-blind inside the top K. Rank-aware metrics reward early hits.
  4. 4. Which metric is the natural choice when relevance has grades such as purchase vs click?

    NDCG sums graded gains with a position discount.

Summary

  • MRR averages 1 / rank of the first relevant item; MAP averages precision at every hit.
  • DCG discounts each gain by log2(position + 1); NDCG divides by the ideal DCG to land in 0 to 1.
  • Position-aware metrics separate rankings that precision and recall at K cannot.
  • Pick the metric from the task (single answer, many hits, graded relevance) and always report the cutoff.
  • Compare with confidence intervals, not the last decimal place.

Next: Beyond Accuracy: Coverage, Diversity and Novelty