Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 36 ยท Evaluation

Retrieval Metrics: Recall, Precision, MRR and nDCG

Measure retrieval quality with hit rate, recall@k, precision@k, MRR and nDCG. Implement each in Python and compare two retrievers on the same labelled queries.

"It feels better" is not a way to improve a RAG system. Changing the chunk size, the embedding model, the hybrid weights or the re-ranker each shifts results in ways that no one can judge by eyeballing three queries. Evaluation gives you numbers so you can compare variants, catch regressions and decide what to ship.

RAG has two parts to evaluate. This lesson covers retrieval (did we fetch the right chunks?), which is cheap, deterministic and fast to run. The next lesson covers the generated answer.

What you need: labelled queries

A retrieval evaluation set is a list of queries, each with the ids of the chunks (or documents) that are relevant. Relevance can be binary (relevant or not) or graded (3 = perfect, 2 = good, 1 = partial, 0 = no). You learn how to build one in Building an Evaluation Set. Here we use a tiny hand-made one.

The core metrics

Let a query have a set of relevant ids, and the system return a ranked list.

Metric Question it answers Formula (per query)
Hit rate@k Is at least one relevant chunk in the top k? 1 if yes, else 0
Recall@k What fraction of the relevant chunks did we find in the top k? found / total relevant
Precision@k What fraction of the top k is relevant? relevant in top k / k
MRR How early does the first relevant chunk appear? 1 / rank of the first relevant (0 if none)
nDCG@k How good is the whole ranking, with graded relevance and position discount? DCG / ideal DCG

Averages over all queries give the reported numbers (for MRR, the "mean" in mean reciprocal rank).

Which to watch:

  • Recall@k and hit rate matter most for RAG, because the model can only use what is in the prompt, and it can ignore a few irrelevant chunks but cannot invent missing ones.
  • MRR and nDCG matter when ordering matters: a re-ranker, a UI that shows ranked results, or a small context budget.
  • Precision@k matters when noise in the prompt hurts (long context, small models).

Implementing the metrics

Python Recall, precision, MRR and nDCG

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check by hand: the first relevant chunk (d2) is at rank 2, so MRR is 0.5. Within the top 3 only d2 is relevant, so recall@3 is 1/2, precision@3 is 1/3. By rank 5 both relevant chunks are found, so recall@5 is 1.

Comparing two retrievers

Now put the metrics to work. Below, two retrievers answer the same labelled queries: plain word-count overlap and TF-IDF. The only difference between the runs is the retriever, so the metric gap is attributable to it.

Python Compare retrievers on a labelled set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both numbers move together here, but in real projects they diverge: a re-ranker may leave recall@50 unchanged while boosting MRR and nDCG at 5, because it only reorders candidates.

Document-level versus chunk-level labels

Labelling the exact chunk ids breaks whenever you change the chunking, so every re-chunking experiment invalidates your set. Two ways around this:

  • Label at document or section level and count a hit if any returned chunk comes from a relevant document. Robust to chunking changes, but coarser.
  • Label with answer spans (the text that answers the question) and count a chunk relevant if it contains the span. Works across chunkings with a simple substring or fuzzy match.

Pitfalls

  • Tiny sets: with 10 queries, one flip changes the metric by 10 points. Aim for 100+ queries for decisions, and report the number of queries.
  • Leakage: do not tune on the same queries you report. Hold out a test portion.
  • Unlabelled relevant chunks: if a retriever finds a relevant chunk that your labels missed, it is scored as wrong. Review misses before trusting low scores.
  • Averages hide failures: look at the worst queries and group results by query type (keyword-like, paraphrase, multi-part).
  • Metric does not equal answer quality. Retrieval metrics are necessary, not sufficient; check end-to-end too.

Keep your evaluation script in version control and run it in CI. A change that drops recall@5 by more than a set margin should fail the build, just like a failing unit test.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. The first relevant chunk is at rank 4. What is that query's reciprocal rank?

    Reciprocal rank is 1 divided by the rank of the first relevant result: 1/4 = 0.25.
  2. 2. Which metric uses graded relevance and discounts lower positions?

    nDCG sums graded gains with a logarithmic position discount and normalizes by the ideal ranking.
  3. 3. Why is recall@k especially important for RAG?

    If the relevant chunk is not in the prompt, generation cannot recover it.
  4. 4. Why do document-level labels survive re-chunking better than chunk-id labels?

    Re-chunking creates new chunk ids, while document ids remain stable.

Summary

  • Retrieval evaluation needs labelled queries and a few metrics you can compute in seconds.
  • Hit rate and recall@k tell you whether the evidence is available; MRR and nDCG tell you about ordering.
  • Compare variants by changing one thing at a time, with enough queries and a held-out set.
  • Inspect failures by query type, not just averages.

Next: Evaluating Generation: Faithfulness and Answer Quality