Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 36 ยท Retrieval Strategies

Re-ranking and Maximal Marginal Relevance

Learn the two-stage retrieve-then-rerank pattern, how cross-encoders differ from bi-encoders, and how MMR removes near-duplicate chunks from RAG context.

First-stage retrieval has to be fast, so it scores a query against millions of chunks using precomputed vectors. That speed costs accuracy: the query and each chunk were embedded separately, and the model never saw them side by side. A re-ranker fixes this. It takes the top 20 to 100 candidates from the first stage and scores each (query, chunk) pair jointly, which is slower but much more precise. You then keep only the top 3 to 6.

A second problem is redundancy: the top five results may be five near-copies of the same paragraph. Maximal marginal relevance (MMR) rebalances relevance against diversity. This lesson covers both.

Retrieve, then re-rank

Stage 1 Fast: 50 candidates
Stage 2 Slow, precise re-rank
1 / 2

Bi-encoders versus cross-encoders

Bi-encoder (embeddings) Cross-encoder (re-ranker)
Input Query and chunk encoded separately Query and chunk encoded together
Chunk vectors Precomputed, stored in the index None; computed per request
Cost per query One embedding plus an ANN lookup One model pass per candidate
Quality Good Better, especially on subtle questions
Used for Stage 1 over the whole corpus Stage 2 over tens of candidates

Popular re-rankers include open cross-encoder models (for example the BGE and MiniLM families), late-interaction models such as ColBERT, and hosted rerank APIs (Cohere, Voyage, Jina and others). A language model can also be prompted to score candidates, at higher cost.

We cannot run a neural cross-encoder in the browser, so the stand-in below scores each (query, chunk) pair jointly with simple features: how many query terms the chunk covers, and whether query word pairs appear next to each other. A real cross-encoder learns far richer signals, but the pipeline is identical: take candidates in, return a new order out.

Python Two-stage retrieval with a pair-wise re-ranker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The re-ranker rewards chunks that contain the phrase "refund for an annual plan" or close variants, and demotes chunks that merely share a word or two. In practice you also read the re-ranker's score as a better-calibrated relevance signal than first-stage similarity, so thresholds work better after re-ranking.

Practical re-ranking advice

  • Candidate count: 20 to 50 candidates is typical. Latency scales with this number, so measure.
  • Truncate chunks to the model's limit (often 512 tokens) before scoring.
  • Batch the pairs in one call; never score them one at a time over the network.
  • Always keep a fallback: if the re-ranker times out, use the first-stage order.
  • Evaluate: compare nDCG and recall@k with and without re-ranking (retrieval metrics). Re-ranking helps most when the first stage already finds the answer but ranks it low.

Redundancy and MMR

Imagine five chunks that all say, in slightly different words, "refunds take 5 business days". They fill your context window and crowd out the one chunk about the exception. MMR picks results one by one. At each step it chooses the candidate maximizing:

lambda * relevance(candidate, query) - (1 - lambda) * max similarity(candidate, already_selected)

  • lambda = 1 is plain relevance ranking.
  • lambda near 0.5 balances relevance and novelty.
  • Lower values emphasize diversity.
Python MMR removes near-duplicates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With lambda = 1.0 MMR equals plain ranking and returns the three near-identical chunks. With 0.5 it keeps one "5 business days" chunk and brings in a different one (gift cards or annual plans). Whether that is good depends on the task: for a question with one precise answer, diversity may add noise, so tune lambda on your evaluation set. Several vector stores and frameworks offer MMR as a built-in search mode.

Other ways to reduce duplication

  • Deduplicate at ingestion with exact hashes and near-duplicate detection (chunking and ingestion covers this).
  • Collapse by document: allow at most two chunks per source document.
  • Merge neighbours: if chunks 7 and 8 of the same page both match, send them as one contiguous passage.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a cross-encoder re-ranker not used over the whole corpus?

    Pairs cannot be precomputed, so cost grows with the number of candidates; that is why it follows a cheap first stage.
  2. 2. What problem does MMR address?

    MMR penalizes candidates similar to those already selected.
  3. 3. In MMR, what does lambda = 1 give you?

    With lambda = 1 the redundancy term has zero weight, so only relevance counts.
  4. 4. A re-ranker times out in production. What is a sensible fallback?

    Degrade gracefully: first-stage results are still useful, just less precise.

Summary

  • Two-stage retrieval: a fast wide first stage, then a slower precise re-ranker over a few dozen candidates.
  • Cross-encoders score the pair jointly and beat bi-encoders on precision.
  • MMR trades relevance against similarity to already-chosen chunks to cut redundancy.
  • Tune candidate count and lambda on your own evaluation set, and keep a fallback.

Next: RAG Prompt Templates and Context Assembly