Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 36 ยท Retrieval Strategies

Retrieval Basics, Top-k and Score Thresholds

Learn how a RAG retriever turns a question into ranked chunks, how to choose top-k, why score thresholds matter, and how to detect questions with no answer.

Retrieval is the first half of retrieval-augmented generation (RAG). Given a user question, the retriever must return a small set of chunks that contain the answer. The language model can only be as good as what you hand it: if the right chunk is missing, even the best model will guess or refuse.

In the earlier chapters you built embeddings, indexes and chunks. This chapter is about the strategy that sits on top of them: how many results to fetch, how to score and combine them, how to filter, and how to re-order. We start with the two knobs every retriever has: k and a score threshold.

The retrieval step

Question The user's words
Embed Turn it into a vector
Search Nearest chunks
Top-k Keep the best few

Everything after this step only sees what retrieval returns.

1 / 2

A tiny retriever to experiment with

Real systems use neural embeddings. In this browser we cannot download an embedding model, so the examples use TF-IDF vectors from scikit-learn as a stand-in. The mechanics (vectors, cosine similarity, top-k) are exactly the same; only the quality of the "meaning" captured is lower. Everything you learn about k, thresholds, filtering and re-ranking transfers directly.

Python A retriever with top-k

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first question uses words ("money back") that appear nowhere in the refund document, so a lexical stand-in struggles. A neural embedding model would link "money back" to "refunds". Keep that gap in mind: it is the main reason hybrid search exists.

Choosing k

k is the number of chunks you pass forward. It is a trade-off:

  • Too small: the answer may sit in rank 4 and you never see it (low recall).
  • Too large: you stuff the prompt with noise, pay for extra tokens, and models become worse at finding the relevant sentence in a long context.

A common recipe is two stages: fetch a generous k (say 20 to 50) cheaply, then use a re-ranker to keep the best 3 to 6 for the prompt. When you have no re-ranker, start with k between 3 and 5 and measure.

You can measure it. If you know which chunk answers each test question, recall@k is the fraction of questions whose right chunk appears in the top k. The next example sweeps k.

Python Recall@k on a tiny test set

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Recall can only go up as k grows, but with diminishing returns. Pick the smallest k whose recall you can live with, then let a re-ranker or the prompt handle the rest.

Score thresholds and "no answer"

Nearest-neighbour search always returns something, even for a question about the weather in a support bot. A score threshold drops weak matches. If nothing survives, the system should say "I don't know" (or hand off to a human) instead of letting the model improvise from irrelevant text.

Python Thresholds and out-of-scope questions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Thresholds are model-specific. Cosine scores from one embedding model are not comparable to another's, and they shift with your corpus. Never copy a number from a blog post: look at the score distribution of known-good and known-bad queries from your own evaluation set (Building an Evaluation Set) and put the cut between them.

Log the top score for every production query. A sudden drop in the average top score is an early warning that users are asking about things your corpus does not cover.

Relative thresholds and score gaps

Absolute thresholds can be brittle. Two alternatives are common:

  • Relative cutoff: keep results whose score is at least, say, 80% of the best score. Good when the best hit is great and the rest is clutter.
  • Elbow or gap detection: cut at the biggest drop between consecutive scores.

Combine them with a hard minimum so a list of uniformly poor matches is still rejected.

Distance versus similarity

Vector databases report either a similarity (higher is better, cosine similarity) or a distance (lower is better, cosine distance = 1 - similarity, or Euclidean distance). Check which one yours returns before you write a threshold, because inverting it is a classic silent bug: you would then keep the worst results.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does recall@k measure in retrieval?

    Recall@k checks whether the known-relevant chunk is among the first k results for each test question.
  2. 2. Why use a minimum score threshold?

    Nearest-neighbour search always returns neighbours; a threshold lets you reject weak ones and fall back.
  3. 3. Why should you not copy a threshold value such as 0.75 from a tutorial?

    Scores are not comparable across models, so calibrate on your own labelled queries.
  4. 4. A vector store returns cosine distance. Which result is the best?

    Distance is lower for closer vectors, so smaller is better; similarity is the opposite.

Summary

  • Retrieval decides what the model can see; generation cannot recover a missing chunk.
  • k trades recall against noise and cost; fetch wide, then re-rank down.
  • Measure recall@k on labelled questions rather than guessing.
  • Use thresholds (absolute, relative or gap-based) to reject weak matches and enable "I don't know".
  • Know whether your store returns similarity or distance.

Next: BM25 and Keyword Search