Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 36 ยท Retrieval Strategies

BM25 and Keyword Search for RAG

Build BM25 from scratch in Python, learn what k1 and b do, and see why exact keyword matching still beats embeddings for codes, names and rare terms.

Embeddings are good at meaning, but they are surprisingly bad at exact strings: error codes, product SKUs, function names, people's surnames. A question like "what does error E-4012 mean?" needs a retriever that treats E-4012 as a rare, important token. That is what keyword search gives you, and the standard algorithm behind it (used by Elasticsearch, OpenSearch, Lucene, SQLite FTS5 and most "sparse" vector features) is BM25.

In this lesson you will implement BM25 in about twenty lines, understand its two parameters, and learn when to prefer it over dense vectors. The next lesson combines both in hybrid search.

The idea in three sentences

BM25 scores a document for a query by summing a score per query term. A term scores higher when it is rare in the corpus (inverse document frequency, IDF) and when it appears often in this document, but the benefit of repeating a word saturates quickly, and long documents are penalized so they do not win just by containing more words.

For a term t in document d:

score(t, d) = IDF(t) * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(d) / avglen))

with IDF(t) = ln(1 + (N - n_t + 0.5) / (n_t + 0.5)), where N is the number of documents and n_t how many contain t.

  • k1 (typically 1.2 to 2.0) controls term-frequency saturation: higher means repeated words keep helping for longer.
  • b (0 to 1, typically 0.75) controls length normalization: 0 ignores length, 1 fully normalizes.

BM25 from scratch

Python Implementing BM25

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The query E-4012 lands on the right document because that token appears in exactly one chunk, so its IDF is high. A dense embedding model has no reliable way to tell E-4012 from E-4021.

What k1 and b really do

Try changing the parameters and watch the scores move. The next example builds two documents: a short one that mentions "export" once, and a long one that mentions it many times among filler.

Python Effect of k1 and b

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output like this:

  • With b = 0 the long document is not penalized, so its extra mentions win easily.
  • With a small k1 repeated mentions barely help, so the short, focused document does better.
  • Defaults (k1 about 1.2 to 1.5, b 0.75) are a reasonable start. If your chunks all have similar length, b matters little.

Tokenization decides everything

BM25 only sees tokens, so preprocessing is part of the retriever:

  • Lowercase and strip punctuation, but keep hyphenated codes and identifiers intact (as above).
  • Stemming or lemmatization (running, runs, ran) raises recall for prose but can hurt code and names.
  • Stop words matter little for BM25 because IDF already shrinks them.
  • Code: split camelCase and snake_case into parts as well as keeping the whole identifier.
  • Multilingual: use a tokenizer per language; whitespace splitting fails for Chinese and Japanese.

Dense versus sparse: who wins where

Question type Better retriever Why
"money back" when docs say "refund" Dense Synonyms and paraphrase
"E-4012", "RFC 9110", a surname Sparse (BM25) Exact rare tokens
Long natural-language question Dense (often) Whole-sentence meaning
Short keyword query Sparse (often) Few tokens, high precision
Domain jargon the embedding model never saw Sparse No training needed

Neither dominates, which is why most production systems run both. Learned sparse models such as SPLADE sit in between: they produce sparse vectors with expansion terms, and several vector databases can store them next to dense vectors.

BM25 inside real systems

You rarely implement BM25 yourself in production. PostgreSQL full-text search, SQLite FTS5, OpenSearch, Elasticsearch, Meilisearch and Typesense provide it, and several vector databases offer sparse or BM25 fields. The Python package rank_bm25 is handy for prototypes. What matters is that you understand the knobs and can evaluate them (retrieval metrics).

BM25 statistics (N, document frequencies, average length) are corpus-wide. When you add or remove many documents, those numbers shift. Most engines recompute them for you, but a hand-rolled index must update them as well.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which problem is BM25 especially good at solving compared with dense embeddings?

    Rare exact tokens get a high IDF and match literally; embeddings blur them.
  2. 2. What does the b parameter control?

    b ranges from 0 (ignore length) to 1 (fully normalize by length).
  3. 3. What does increasing k1 do?

    k1 sets term-frequency saturation; a higher value delays the plateau.
  4. 4. Why is IDF used?

    A term that appears in few documents discriminates between them better, so it gets a bigger weight.

Summary

  • BM25 scores documents by rare-term matches with saturating term frequency and length normalization.
  • k1 (saturation) and b (length normalization) rarely need much tuning; start at about 1.5 and 0.75.
  • Tokenization choices affect results as much as the formula.
  • BM25 wins on exact identifiers and jargon; embeddings win on paraphrase.
  • Real systems combine both.

Next: Hybrid Search and Reciprocal Rank Fusion