Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 53 ยท Evaluation

Metrics: Retrieval, Benchmarks and Statistics

Compute precision and recall at k, MRR and nDCG for retrieval, understand benchmark contamination, and put confidence intervals around your eval scores.

A retrieval system returns a ranked list, and "is it good?" has no single answer. Did the right document appear at all? Was it first or fifth? Did the junk crowd out useful results? Different metrics answer different questions, and choosing the wrong one can hide a problem.

This lesson covers the standard ranking metrics (precision@k, recall@k, MRR, nDCG), explains what public benchmarks can and cannot tell you, and shows how to attach a confidence interval to a score so you know whether a difference is real. Retrieval itself is covered in Vector Retrieval in Practice and Hybrid Search and Reranking.

Setting up: relevant documents and a ranking

For each test query you need the set of relevant documents (judged by people, or derived from known answers). The system returns a ranked list. Metrics compare the two. In the examples below, ranked is a list of document ids and relevant is a set of ids.

Precision@k and recall@k

  • Precision@k: of the top k results, what fraction is relevant? It measures how clean the list is.
  • Recall@k: of all relevant documents, what fraction appears in the top k? It measures coverage.
TEXT
precision@k = relevant in top k / k
recall@k    = relevant in top k / total relevant

In RAG, recall@k often matters most, because the generator can only use what was retrieved. If the answer-bearing chunk is not in the top k, no prompt can save you. Precision matters too, since irrelevant chunks waste the context budget and can distract the model.

MRR: how soon is the first good result?

Reciprocal rank is 1 divided by the position of the first relevant result (1 for first place, 0.5 for second, 0.33 for third, 0 if none). The mean reciprocal rank (MRR) averages it over queries. It suits cases where one good answer is enough, such as question answering.

nDCG: graded relevance and position

Sometimes relevance is graded: 3 = perfect, 2 = good, 1 = marginal, 0 = irrelevant. Discounted cumulative gain (DCG) adds up the grades but discounts lower positions:

TEXT
DCG@k = sum over positions i = 1..k of  grade_i / log2(i + 1)

nDCG divides DCG by the DCG of the ideal ordering, so it lies between 0 and 1 and can be compared across queries. A result at position 1 counts fully, at position 3 it counts half (log2(4) = 2), and so on.

Which metric answers what?

Precision@k How clean is the top k?
Recall@k Did we find what exists?
MRR How soon is the first hit?
nDCG Graded and position-aware.

For RAG, watch recall@k first, then ranking quality.

1 / 2
Python Ranking metrics from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how q3, a total miss, scores zero on every metric. When you average over queries, a few complete misses can drag the mean down more than many small ranking errors, so also look at the list of zero-score queries, not only the average.

Benchmarks and what they really measure

A benchmark is a public test set with an agreed scoring method. Examples cover broad knowledge questions, grade-school and competition math, code generation with unit tests, reading comprehension and retrieval. They are useful for rough comparison between models, but they have limits:

  • Contamination. If benchmark questions (or their answers) appeared in the training data, a high score may reflect memorization. Models trained on large web crawls can easily have seen popular test sets.
  • Saturation. Once top models score 95% or more, the benchmark cannot separate them.
  • Mismatch. A benchmark is not your task. A model that is best at exam-style multiple choice may not be best at drafting your support replies.
  • Prompt sensitivity. Scores depend on the exact prompt format, number of examples and answer extraction, so numbers from different reports are often not comparable.
  • Gaming. When a leaderboard becomes a target, people optimize for it.

Treat public benchmarks as a shortlist filter, then decide with your own eval set, drawn from your own data, which you keep private.

Confidence intervals for eval scores

If your eval has 50 cases and the system passes 42, the score is 84%. But with a different 50 cases you would get a different number. A confidence interval gives a plausible range for the true pass rate.

A simple, assumption-light method is the bootstrap: resample your cases with replacement many times, compute the score of each resample, and take the middle 95% of those scores. For pass/fail data a formula also works: the standard error is sqrt(p * (1 - p) / n), and a rough 95% interval is p ยฑ 1.96 * standard error.

To compare two systems on the same cases, compute the paired difference per case (1 if A passes and B fails, -1 for the reverse, 0 otherwise) and bootstrap that. If the interval for the average difference excludes zero, the gap is unlikely to be noise.

Python Bootstrap confidence interval and paired comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With only 50 cases, an interval a dozen points wide is normal. If you need to detect a 2-point improvement, you need hundreds or thousands of cases. Do the arithmetic before you run the experiment, and report intervals, not just point scores.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A query has 4 relevant documents. The top 5 results contain 2 of them. What are precision@5 and recall@5?

    Precision is 2/5 = 0.4 (relevant in top k over k). Recall is 2/4 = 0.5 (relevant found over all relevant).
  2. 2. The first relevant result is at position 4. What is the reciprocal rank for that query?

    Reciprocal rank is 1 divided by the position of the first relevant result: 1/4.
  3. 3. Why can a high benchmark score be misleading?

    Contamination, saturation and task mismatch all limit what a public benchmark tells you.
  4. 4. Your new prompt scores 86% and the old one 82% on 50 cases. What should you conclude?

    A 4-point gap on 50 cases is two flipped questions, which is easily explained by chance.

Summary

  • Precision@k measures a clean top k; recall@k measures coverage and is crucial for RAG; MRR rewards an early first hit; nDCG handles graded relevance and position.
  • Look at per-query results and complete misses, not only averages.
  • Public benchmarks are affected by contamination, saturation and prompt sensitivity; use them to shortlist, then test on your own private data.
  • Report eval scores with confidence intervals (bootstrap or formula) and compare systems with paired differences.
  • Plan the number of cases from the smallest difference you care about.

Next: see why models invent facts in Hallucinations and How to Reduce Them.