Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 43 ยท How Search Works

Relevance Scoring with BM25

Learn how Elasticsearch scores matches with BM25: term frequency, inverse document frequency, field length, k1 and b, boosts and _explain, with Python code.

A search engine does not only decide whether a document matches; it decides how well. Each hit gets a number called the score (_score in the response), and results are sorted by it, highest first. Since version 5, Elasticsearch scores text with an algorithm called BM25 ("Best Matching 25"), which replaced the older TF-IDF scoring.

BM25 rests on three plain-language observations. A word that appears more often in a document is a better sign of relevance. A word that is rare across all documents says more than a common one. And a match in a short field means more than the same match buried in a long one. This lesson turns those three ideas into a formula, shows the _explain output of the real engine (read-only), and lets you compute BM25 yourself with a Python stand-in that you can run here.

The three ingredients

Term frequency (tf) is how many times the query term appears in the field of this document. Ten mentions of "espresso" suggest a document is about espresso. But BM25 applies diminishing returns: the tenth mention adds far less than the first. Old TF-IDF grew without a limit, which let keyword stuffing win.

Inverse document frequency (idf) measures rarity. If every document contains "the", then "the" tells you nothing, and its idf is close to zero. If one document in a million contains "grinder", that word is a strong clue and its idf is high. In Lucene's BM25:

Text
idf = ln(1 + (N - n + 0.5) / (n + 0.5))

where N is the number of documents and n the number that contain the term.

Field length normalization compares the length of this document's field to the average length. If a term occurs once in a three-word title, the title is about that term. Once in a 300-word description, it is a passing mention. Shorter than average gives a boost; longer than average a penalty.

The BM25 formula

For one term in one document, Lucene computes:

Text
score = idf * tf / (tf + k1 * (1 - b + b * dl / avgdl))

where dl is the field length of the document (in tokens) and avgdl is the average length across the index. For a query with several terms, the per-term scores are added up.

Two tunable parameters shape it:

  • k1 (default 1.2) controls term-frequency saturation. With k1 = 0, only the presence of a term matters, no matter how often it occurs. A larger k1 lets repeated occurrences keep adding to the score for longer.
  • b (default 0.75) controls length normalization. b = 0 ignores field length entirely; b = 1 applies full normalization.

Most indexes keep the defaults. Change them only when you can measure the effect on real queries. They are set per field in the mapping through a custom similarity (see Mappings and Field Types).

One honest simplification: Elasticsearch stores the field length in a compressed one-byte form, so the real dl is an approximation of the true length, and scores are computed per shard, with that shard's own statistics. On a tiny index spread over several shards, idf can differ between shards, so scores in small test data may look odd. Production indexes are large enough for this to average out.

What makes a score high

Term frequency More occurrences help, with diminishing returns.
Rare terms Uncommon words count more (idf).
Short fields A match in a short field wins.
k1 and b Tune saturation and length effect.
1 / 2

A BM25 scorer in Python

The stand-in below implements the formula above for a five-document corpus. It also computes a plain TF-IDF score (tf * ln(N / n)) so you can compare the two. It is a teaching model with a simple tokenizer, not the engine's code; real Elasticsearch works with analyzed terms and Lucene's data structures.

Python BM25 versus TF-IDF on a small corpus

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Running it prints:

TEXT
docs: 5 average length: 6.4
espresso  df=2 idf=0.875
coffee    df=2 idf=0.875
machine   df=2 idf=0.875
kettle    df=1 idf=1.386
query: coffee machine {}
  doc3 len=18 bm25=0.725 tfidf=3.665
  doc2 len= 3 bm25=0.508 tfidf=0.916
  doc1 len= 6 bm25=0.408 tfidf=0.916
query: espresso {}
  doc1 len= 6 bm25=0.557 tfidf=1.833
  doc4 len= 3 bm25=0.508 tfidf=0.916

Look at the idf lines first: kettle occurs in one document, so it is worth more (1.386) than words found in two. For "coffee machine", document 3 contains both words twice, so it wins in both systems. The gap is the interesting part: TF-IDF gives it 3.665 against 0.916, four times more, while BM25 gives 0.725 against 0.508, because repeated words saturate. In the "espresso" query, document 1 mentions the word twice, yet it beats the short document 4 only narrowly under BM25 (0.557 against 0.508): the second mention adds little, and document 4's short length helps it.

Tuning k1 and b

Now change the parameters and watch the ranking move. With b = 0 the long document is no longer punished for its length. With a very small k1, repeated words stop helping, so only which terms match counts.

Python What k1 and b do

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output:

TEXT
defaults k1=1.2 b=0.75   ['doc3=0.72', 'doc2=0.51', 'doc1=0.41']
no length norm b=0       ['doc3=1.09', 'doc1=0.40', 'doc2=0.40']
full length norm b=1     ['doc3=0.65', 'doc2=0.56', 'doc1=0.41']
tiny k1=0.1              ['doc3=1.57', 'doc2=0.83', 'doc1=0.80']
large k1=5               ['doc3=0.25', 'doc2=0.22', 'doc1=0.15']
-- query 'espresso'
defaults                 ['doc1=0.56', 'doc4=0.51']
tiny k1=0.1              ['doc1=0.84', 'doc4=0.83']

With b=0 the short document 2 loses its length advantage and ties with document 1 at 0.40 (they now differ only by idf and tf, which are equal), while the long document 3 jumps from 0.72 to 1.09. With b=1 the short document gets the biggest boost. With a tiny k1 the score is close to "how many query terms matched": document 3 matches both terms and scores about twice the others, and the second espresso in document 1 barely counts (0.84 against 0.83). A large k1 lowers every score here, because the length normalizer sits in a bigger denominator; the order stays the same, which shows that tuning k1 and b mostly shifts the details.

Reading _explain

When a ranking surprises you, ask Elasticsearch to show its arithmetic. The _explain API explains the score of one document for a query:

JSON
GET /products/_explain/1
{
  "query": { "match": { "title": "espresso" } }
}

The answer is a tree. Abbreviated, and with illustrative numbers, it looks like this:

JSON
{
  "matched": true,
  "explanation": {
    "value": 1.33,
    "description": "weight(title:espresso in 0) [PerFieldSimilarity], result of:",
    "details": [
      {
        "value": 1.33,
        "description": "score(freq=2.0), computed as boost * idf * tf from:",
        "details": [
          { "value": 2.2,  "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:" },
          { "value": 0.60, "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:" }
        ]
      }
    ]
  }
}

You will recognize the pieces: freq, the idf formula, and the tf part with k1, b, dl and avgdl. The numbers will differ on your data. You can also add "explain": true to a _search request to get the tree for every hit. It is slow, so use it only for debugging.

Choosing a similarity per field

BM25 is the default, but the scoring model is configurable per field. You define named similarities in the index settings when you create the index, then point fields at them in the mapping:

JSON
PUT /articles
{
  "settings": {
    "index": {
      "similarity": {
        "short_text_bm25": { "type": "BM25", "k1": 1.5, "b": 0.25 },
        "dirichlet":       { "type": "LMDirichlet", "mu": 2000 }
      }
    }
  },
  "mappings": {
    "properties": {
      "title":    { "type": "text", "similarity": "short_text_bm25" },
      "body":     { "type": "text" },
      "keywords": { "type": "text", "similarity": "boolean" }
    }
  }
}

The built-in choices are:

Type What it does
BM25 The default. Parameters k1, b and discount_overlaps.
boolean The score is only the boost for a match, with no frequency or length effect. Good for tag-like text.
LMDirichlet, LMJelinekMercer Language-model scoring with smoothing (mu, lambda).
DFR, IB, DFI Divergence-from-randomness, information-based and independence models.
scripted Your own formula written in a script.

Two practical rules. First, a field's similarity cannot be changed once documents are indexed; you create a new index and reindex. Second, scores from different similarities are not comparable with each other, so mixing them inside one query needs care. For most applications, tuning k1, b and boosts is enough (the section above).

Boosts

Sometimes you know that a title match matters more than a description match. A boost multiplies a score. You can boost a whole clause, or a field inside multi_match:

JSON
GET /products/_search
{
  "query": {
    "multi_match": {
      "query": "espresso machine",
      "fields": ["title^3", "description"]
    }
  }
}

title^3 multiplies the title's contribution by 3. Boosts are blunt tools: raise them in small steps and check the results, because they interact with idf and length effects. A boost of 1 changes nothing, and a boost between 0 and 1 lowers a clause's weight. More complex scoring ideas appear in Bool Queries and Compound Logic.

The last stand-in applies boosts to a title and a description, and shows how the ordering can flip.

Python Boosting a field

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output:

TEXT
title^1: B=0.51, A=0.32
title^3: A=0.95, B=0.51

Document B only mentions "espresso" in its description (three times), and A only in its title. With no boost, B's repeated mentions win. Tripling the title weight puts A on top. That is the whole purpose of a boost: say which field counts more.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which statement about BM25 term frequency is true?

    The tf part is freq / (freq + something), which flattens out. The parameter k1 controls how fast.
  2. 2. A term appears in almost every document. How does it affect the score?

    Idf measures rarity. Common terms such as the carry little information, so they contribute little.
  3. 3. What does setting b to 0 do?

    b controls length normalization. With b = 0 the dl / avgdl part drops out of the formula.
  4. 4. Which API shows how a score for one document was calculated?

    _explain returns a tree with idf, tf, k1, b, dl and avgdl for the document you ask about.
  5. 5. Why can scores look odd on a tiny test index with several shards?

    Each shard scores with its own document counts. With very little data, those counts differ noticeably between shards.

Summary

  • Elasticsearch ranks text matches with BM25: term frequency with saturation, inverse document frequency, and field length normalization.
  • k1 (default 1.2) sets how quickly repeated terms stop helping; b (default 0.75) sets how much field length matters.
  • Scores only mean something for ranking within one query, and are computed from per-shard statistics.
  • _explain (or "explain": true) shows the arithmetic behind a score; boosts like title^3 multiply a clause.
  • The Python scorer is a stand-in that follows the Lucene formula on a toy corpus, not the engine itself.

Next: Segments, Refresh and Near-Real-Time Search.