Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 38 ยท Retrieval and Context Engineering

Ranking, Deduplicating and Assembling Context

Turn raw retrieval hits into a clean prompt: pick top-k, remove near-duplicates, balance relevance with diversity, order passages and fit the token budget.

Retrieval returns a ranked list. Pasting that list into a prompt unchanged works for demos and fails in production: the top hits may be near-copies of each other, a bad hit may sit first, and the total may not fit the window. Between retrieval and generation sits a step of its own, context assembly.

This lesson walks through the steps in order: cut by relevance, remove duplicates, add diversity, arrange the passages, and fit the budget. Every block is runnable Python; relevance scores are hard-coded so you can see each step's effect.

The assembly pipeline

flowchart LR
    A["Ranked hits"] --> B["Score threshold"]
    B --> C["Remove near-duplicates"]
    C --> D["Diversify (MMR)"]
    D --> E["Order passages"]
    E --> F["Fit token budget"]
    F --> G["Prompt block"]

Each step removes or rearranges material before the model sees it.

Step 1: threshold and top-k

Taking "the top 5" always returns five items, even when only one is relevant. Combine a top-k limit with a minimum score so weak matches are dropped. If nothing passes, the prompt takes the "no information" path from RAG basics.

Step 2: remove near-duplicates

Documents often repeat themselves: the same policy in two versions, a page and its printable copy. Duplicates waste budget and make the model overweight one fact. A simple test is the word-set overlap (Jaccard similarity) between two passages.

Python Drop near-duplicate passages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exact word overlap misses paraphrases. Production systems compare embeddings for the same purpose; the decision logic stays identical.

Step 3: relevance plus diversity

If the top five passages all describe the same sub-topic, the sixth, which covers another aspect, is more valuable than a fifth copy. Maximal Marginal Relevance (MMR) picks passages one at a time, trading relevance to the question against similarity to what is already chosen:

mmr_score = lambda * relevance - (1 - lambda) * max_similarity_to_selected

Python MMR: pick relevant but different passages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Plain top-3 returns the same price twice. MMR swaps the duplicate for the student discount. Raise lam toward 1 for pure relevance; lower it for more variety.

Step 4: order the passages

Models use the start and end of long inputs better than the middle. Two reliable layouts:

  • Best last: put the most relevant passage nearest the question, which is a good default for short contexts.
  • Edges: put the strongest passages at the start and the end and the weakest in the middle.

Whichever you pick, keep it consistent and measure; the effect depends on the model. When sources have dates, sorting by date and saying "newer passages override older ones" in the prompt resolves conflicts.

Step 5: fit the budget and format

Add passages in priority order until the budget from the token lesson is used. Never cut a passage in the middle; skip it or use a shorter one. Then wrap each passage with an id and metadata so the model can cite it.

Python Fit passages into a budget and format them

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Query rewriting

Retrieval quality depends on the query. A chat message like "and for students?" means nothing alone. Before retrieval, rewrite it using the history ("What discounts exist for students on the premium plan?"). The rewrite is itself a small prompt: give the history and the last message and ask for one standalone question. Retrieve with the rewritten query, but show the original to the user.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why combine top-k with a minimum score?

    A threshold lets the pipeline return fewer passages, or none, when matches are weak.
  2. 2. What does MMR optimize?

    MMR penalizes passages that repeat what is already selected, adding diversity.
  3. 3. Where do models tend to use long input least reliably?

    Details in the middle of a long context are more likely to be missed than those at the edges.
  4. 4. Why rewrite a chat follow-up before retrieval?

    A standalone question gives the retriever the information it needs to find the right passages.

Summary

  • Assembly sits between retrieval and generation: threshold, deduplicate, diversify, order, fit.
  • Duplicates waste budget; MMR trades relevance for variety.
  • Order matters: put the strongest evidence at the edges of the context.
  • Never truncate a passage mid-way; skip it or use a shorter one.
  • Rewrite conversational queries into standalone questions before retrieving.

Next: Conversation Memory and Summarization