Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 53 ยท Retrieval-Augmented Generation (RAG)

Hybrid Search and Reranking

Combine BM25 keyword search with vector search using reciprocal rank fusion, then improve results with rerankers, query rewriting and HyDE. Live Python.

Vector search finds text with the same meaning, even when the words differ. Keyword search finds text with the same words, even when the meaning is hard to embed. Each fails where the other succeeds: a vector search for an error code like E4021 may return vaguely related error pages, while a keyword search for "get my money back" will never find the page that says "refund".

Production RAG systems therefore rarely rely on one retriever. They run both, merge the lists (hybrid search), and often put a slower but smarter reranker on top. This lesson builds the pieces: BM25, reciprocal rank fusion, reranking, query rewriting and HyDE. It assumes you have read Vector Retrieval in Practice.

Two retrievers, one list

Keyword (BM25) Exact words, names, codes, rare terms.
Vector Paraphrases and related meaning.
Fusion Merge both rankings into one.
Reranker Re-score the best candidates carefully.

Cast a wide net cheaply, then judge the catch carefully.

1 / 2

BM25: keyword search done well

BM25 is the classic ranking function behind search engines such as Lucene, Elasticsearch and OpenSearch. For each query word that appears in a document it adds a score built from three ideas:

  • Term frequency: more occurrences of the word help, but with diminishing returns (the tenth "refund" matters much less than the first).
  • Inverse document frequency (IDF): rare words count more. "the" says nothing; "E4021" says a lot.
  • Length normalization: a match in a short document is stronger evidence than the same match in a very long one.

Two parameters tune it: k1 controls how fast term frequency saturates (commonly 1.2 to 2), and b controls how strongly length is normalized (commonly 0.75). BM25 needs no training, no embeddings and handles exact identifiers perfectly. Its weakness is vocabulary mismatch: no shared word, no match.

Reciprocal rank fusion

How do you merge a BM25 list and a vector list? You cannot add the scores: BM25 scores might range from 0 to 15, cosine similarities from 0 to 1, and the scales differ per query. Reciprocal rank fusion (RRF) sidesteps the problem by using only ranks:

TEXT
rrf_score(doc) = sum over retrievers of 1 / (k + rank_of_doc_in_that_retriever)

A document ranked first by both retrievers scores highest; a document found by only one still gets credit. The constant k (usually 60) stops the very top ranks from dominating. RRF needs no score normalization and works with any number of retrievers, which is why most hybrid systems use it first.

The example below implements BM25 from scratch, a small stand-in for a dense (vector) retriever, and RRF. The stand-in merges synonyms into shared concepts, like an embedding model would, but it knows nothing about identifiers.

Python BM25, a dense stand-in and reciprocal rank fusion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the three queries as three lessons. E4021 is found only by the keyword side, because the dense stand-in has never seen the token (real embedding models are also weak on rare identifiers). "get my money back" is found by both, so d3 gets the strongest fused score. In "card declined", both retrievers agree on the top two, and the dense list adds a third candidate. Fusion keeps what either retriever is good at.

Reranking

First-stage retrievers must be fast, because they score the whole corpus. Vector search compares the query and each document separately: each side is squeezed into one vector before they ever meet. That loses detail.

A cross-encoder reranker reads the query and one candidate together in a single pass of a transformer and outputs a relevance score. It can notice that a document mentions the right word but about a different product, or that it answers a question rather than just sharing its vocabulary. It is much more accurate and far too slow to run over millions of documents, so the usual pipeline is:

  1. Retrieve 50 to 100 candidates cheaply (hybrid search).
  2. Rerank those candidates with a cross-encoder (an API such as a hosted reranker, or a small open model).
  3. Keep the top 3 to 8 for the prompt.

Rerankers add latency (tens to hundreds of milliseconds) and cost, so measure whether they improve your metrics (see Metrics: Retrieval, Benchmarks and Statistics) before keeping one.

Query rewriting and HyDE

Users type short, messy questions, and the best search query is often not the user's literal text.

  • Query rewriting: ask an LLM to turn the question into a better search query, expand abbreviations, add synonyms, or use the conversation history ("what about the second one?" becomes "return policy for the second item").
  • Multi-query: generate several rewrites, retrieve for each, and fuse the lists with RRF.
  • HyDE (Hypothetical Document Embeddings): ask the model to write a fake answer to the question and search with that text. A short question and a long answer-shaped passage live in different parts of embedding space, and an answer-shaped query lands closer to real answer passages. The invented answer may contain false facts, so it is used only for searching and is never shown or trusted.
  • Query decomposition: split a complex question into sub-questions, retrieve for each, and merge.

All of these cost an extra model call before retrieval. They help most on short, vague or conversational queries.

The next example uses a scripted stand-in for the model calls. The "cross-encoder" scores each query-and-document pair with simple pair features (shared words, a phrase bonus), which is the shape of the real thing even though it is not a neural network.

Python Reranking, query rewriting and HyDE with stand-ins

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The literal question "where is my cash" shares no words with any document, so every score is zero. The rewrite and the hypothetical answer both contain the vocabulary of the real answer, so the right document rises to the top. The same trick works for real retrievers, with the difference that a model writes the rewrite.

Do not add every technique at once. Build a small test set of real questions with known right chunks, then add one technique at a time and keep it only if the numbers improve.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which query is the strongest argument for adding BM25 to a vector search system?

    Exact identifiers and rare terms are what keyword search handles best, while embeddings often blur them.
  2. 2. Why does reciprocal rank fusion use ranks instead of raw scores?

    Different retrievers produce scores on incompatible scales. Ranks are comparable, so RRF can merge any number of lists.
  3. 3. Why is a cross-encoder reranker run only on a short candidate list?

    Each query-document pair needs its own model pass, so cost grows with the number of candidates.
  4. 4. In HyDE, what happens to the model's hypothetical answer?

    The invented answer only helps find real passages. The final answer must be grounded in the retrieved text.

Summary

  • BM25 matches exact words and rare terms; vector search matches meaning. Hybrid search uses both.
  • Reciprocal rank fusion merges rankings with 1 / (k + rank), no score normalization needed.
  • A cross-encoder reranker re-scores a short candidate list far more accurately than first-stage retrieval, at higher cost per document.
  • Query rewriting, multi-query and HyDE improve short or vague questions; HyDE answers are for searching only.
  • Add each technique only when a test set shows it helps.

Next: turn the retrieved chunks into a grounded answer in Answering with Context and Citations.