Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 43 ยท Searching

More Like This and Similarity Queries

Find related documents with Elasticsearch more_like_this, match a minimum number of terms with terms_set, tune per-field BM25 similarity and build a simple recommender.

"Customers who read this also read..." and "Related articles" are everywhere, and you rarely need machine learning to get a good first version. Elasticsearch can take one document, pick the most characteristic words in it, and search for other documents that share them. That is the more_like_this query (MLT).

This lesson explains how MLT chooses terms and which settings matter, then adds the terms_set query, which matches "at least N of these terms", and shows how to change the similarity function of a field. The cluster cannot run here, so requests are read-only and the runnable examples are JavaScript stand-ins of the ideas.

The more_like_this query

You can give MLT a piece of text, or point it at an existing document:

HTTP
GET /articles/_search
{
  "query": {
    "more_like_this": {
      "fields": ["title", "body"],
      "like": [
        { "_index": "articles", "_id": "42" },
        "tidal energy storage"
      ],
      "min_term_freq": 1,
      "min_doc_freq": 2,
      "max_query_terms": 25,
      "minimum_should_match": "30%"
    }
  }
}

Behind the scenes it works in three steps:

  1. Read the source. For a like document it fetches the stored fields (or analyzes the text you gave). The document itself is excluded from the results.
  2. Pick interesting terms. It scores each term by how often it appears in the source (tf) against how rare it is in the index (idf), and keeps the best max_query_terms. A word like "the" is frequent everywhere and scores low, while "tidal" scores high.
  3. Build a query. The chosen terms become a bool query of should clauses, and minimum_should_match says how many must match.

The settings that matter

Setting Meaning Default
min_term_freq Ignore terms that appear fewer times in the source 2
min_doc_freq Ignore terms found in fewer documents of the index 5
max_doc_freq Ignore terms found in more documents (very common words) unbounded
max_query_terms How many terms the generated query may have 25
minimum_should_match How many generated clauses must match 30%
stop_words Words to skip none
unlike Documents or text whose terms should be avoided none

The defaults are made for big indices. On a small demo index, min_doc_freq: 5 throws away almost everything and the query returns nothing. That is the classic reason for an "MLT returns no results" mystery, and the fix is to lower min_term_freq and min_doc_freq to 1 or 2 while experimenting.

Practical notes

  • Choose fields that carry meaning: title, tags, body. Fields with ids or numbers add noise.
  • Fields must be analyzed text (or keyword) and stored in _source, because MLT re-analyzes the source text when needed.
  • Filter the result. Wrap the query in bool and add a filter to stay within the same language, the same category, or published content only.
  • It is a lexical method. It finds documents with the same words, not the same meaning. For meaning, combine it with vector search.
HTTP
GET /articles/_search
{
  "query": {
    "bool": {
      "must": {
        "more_like_this": {
          "fields": ["title", "tags"],
          "like": [{ "_index": "articles", "_id": "42" }],
          "min_term_freq": 1, "min_doc_freq": 2
        }
      },
      "filter": [
        { "term": { "language": "en" } },
        { "term": { "published": true } }
      ]
    }
  }
}

A term picker you can run

The stand-in below does what step 2 does: it computes a tf-idf score for each word of a source article against a small corpus, picks the top terms and finds the best matching documents. It is not Elasticsearch's exact formula, but the idea is the same.

JS Choosing interesting terms and finding related articles

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last line returns nothing: no term occurs in five of six documents, so the defaults filter out everything. That is exactly the "MLT finds nothing" mistake on a small index.

terms_set: match at least N terms

terms_set is the opposite shape. You give a list of terms and say how many of them a document must contain. The number can come from a field of the document itself, which makes it good for skills matching, permissions and tags:

HTTP
GET /candidates/_search
{
  "query": {
    "terms_set": {
      "skills": {
        "terms": ["java", "kafka", "docker", "sql"],
        "minimum_should_match_field": "required_matches"
      }
    }
  }
}

Here every candidate document stores required_matches, the number of skills the person is looking for, and the query matches those whose skills list contains at least that many of the four terms. Use minimum_should_match_script when the number comes from an expression. Note that skills must be a keyword field, since the terms are matched exactly.

JS terms_set with a per-document threshold

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Chen has three matching skills but needs four, so Chen is not returned, while Bo needs only one.

Per-field similarity

MLT and every other full-text query ends with a similarity, the function that turns term statistics into a score. The default is BM25, with two knobs: k1 (how fast repeated terms stop adding to the score) and b (how much long fields are penalized). You can set different values per field in the mapping, using a named similarity in the index settings:

HTTP
PUT /articles
{
  "settings": {
    "similarity": {
      "short_field_bm25": { "type": "BM25", "k1": 1.2, "b": 0.3 },
      "tag_boolean":      { "type": "boolean" }
    }
  },
  "mappings": {
    "properties": {
      "title": { "type": "text", "similarity": "short_field_bm25" },
      "tags":  { "type": "text", "similarity": "tag_boolean" },
      "body":  { "type": "text" }
    }
  }
}

A boolean similarity gives a flat score for a match and ignores frequency, which suits tags. Lowering b makes field length matter less. Changing a field's similarity needs a new index and a reindex, so decide early and measure with realistic queries. Chapter Relevance and BM25 explains the parameters in detail.

A simple recommender

Putting the pieces together gives a decent "related items" feature: use MLT on title and tags from the viewed item, filter to the same language and to published items, cap the size to a handful, and cache the result for a while. Log which suggestions are clicked and adjust minimum_should_match. When results look random, the usual causes are fields that carry noise, a max_query_terms that is too high, or documents that are too short to have meaningful terms.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A more_like_this query on a small test index returns no hits. What is the most likely cause?

    The defaults suit large indices. On a tiny index few terms occur in enough documents, so no terms are left to search for.
  2. 2. How does more_like_this decide which terms of the source document to use?

    Terms that are frequent in the source but rare in the index are the most characteristic, so they are chosen first.
  3. 3. What does terms_set add compared with a plain terms query?

    terms matches any one of the values. terms_set lets you require a minimum count, even a different one for each document.
  4. 4. Which change needs a reindex?

    Similarity is part of how a field is indexed and scored, so existing fields cannot switch to a new one in place. Query options can change at any time.

Summary

  • more_like_this turns a document or text into a query of its most characteristic terms and returns similar documents.
  • The key settings are min_term_freq, min_doc_freq, max_query_terms and minimum_should_match; defaults can make small indices return nothing.
  • terms_set requires a minimum number of matching terms, fixed or taken from a field of each document.
  • Per-field similarity (BM25 parameters or boolean) is set in the mapping and needs a reindex to change.
  • MLT is word-based; use it with filters, and combine it with vector search for meaning.

Next: learn to match documents the other way round in Percolator and Alerting Patterns.