Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 43 ยท Searching

Highlighting, Autocomplete and "Did You Mean"

Highlight matches, build autocomplete with the completion suggester, search_as_you_type and edge n-grams, and add did-you-mean with term and phrase suggesters.

Good search is more than a ranked list. Users want to see why a result matched (highlighting), they want suggestions while typing (autocomplete), and they want help when they misspell something ("did you mean"). Elasticsearch has a tool for each: the highlight section of a search, the completion suggester and search_as_you_type field for autocomplete, and the term and phrase suggesters for spelling.

This lesson shows each tool, explains when to choose which autocomplete approach, and builds an edge n-gram autocomplete in JavaScript so you can see how prefix matching works inside an index. The real requests are read-only; the stand-in is simplified.

Help the searcher

Highlight Show why a hit matched.
Autocomplete Suggest while the user types.
Did you mean Offer a corrected spelling.

Each one is a different feature with its own cost.

1 / 2

Highlighting

The highlight section returns fragments of a field with the matching words wrapped in tags:

JSON
GET /articles/_search
{
  "query": { "match": { "body": "inverted index" } },
  "highlight": {
    "fields": {
      "body": {
        "fragment_size": 120,
        "number_of_fragments": 2
      }
    },
    "pre_tags": ["<mark>"],
    "post_tags": ["</mark>"]
  }
}

Each hit gets a highlight object next to _source:

JSON
{
  "_source": { "title": "How search works" },
  "highlight": {
    "body": [
      "An <mark>inverted</mark> <mark>index</mark> maps each term to the documents that contain it."
    ]
  }
}

Points to know:

  • The default tags are <em>. Use pre_tags and post_tags for your own markup.
  • Elasticsearch does not escape your text. If the field contains user-generated HTML, set "encoder": "html" so tags in the content are escaped and only your highlight tags stay as markup. Skipping this is a classic XSS bug.
  • Highlighting re-analyzes the stored text (or uses stored offsets) for the top hits only, so it is cheap for a page of ten but not something to do for thousands of hits.
  • The unified highlighter is the default. It picks fragments by score and breaks on sentence boundaries. It reads the original text from _source unless you mapped the field with term_vector or offsets in postings, which speeds up very large fields.
  • highlight_query lets you highlight with a different query than the search, and require_field_match: false highlights matches in fields other than the one that matched.

Highlights come from the query's analysis. If a stemmer turns running into run, searching run highlights running. That is correct behavior, not a bug.

Three ways to build autocomplete

Autocomplete means finding results from a partial word, quickly, on every keystroke. Elasticsearch gives you three main routes.

Approach How it works Strength Weakness
completion suggester Special in-memory structure (FST) of prefixes Extremely fast, great for known suggestion lists Prefix only (with fuzzy option), separate from search
search_as_you_type field Field with built-in shingles and edge n-grams Easy, matches words in any position Larger index, more fields
Custom edge n-gram analyzer You choose the analyzer and limits Most flexible You must configure it carefully

The completion suggester

Map a field with the completion type, and index the phrases people should find:

JSON
PUT /cities
{
  "mappings": {
    "properties": {
      "suggest": { "type": "completion" }
    }
  }
}

PUT /cities/_doc/1
{
  "name": "San Francisco",
  "suggest": {
    "input": ["San Francisco", "SF", "Frisco"],
    "weight": 90
  }
}

Ask for suggestions with the suggest section:

JSON
POST /cities/_search
{
  "suggest": {
    "city_suggest": {
      "prefix": "san f",
      "completion": {
        "field": "suggest",
        "size": 5,
        "skip_duplicates": true,
        "fuzzy": { "fuzziness": 1 }
      }
    }
  }
}

The suggester lives in memory, so it answers in a fraction of a millisecond. It matches from the beginning of an input only: francisco will not match San Francisco unless you added it as another input. It ignores your analyzers' filters apart from the ones configured on the completion field. The weight orders suggestions, which is how you rank by popularity. You can add contexts (for example a category or a location) to filter suggestions.

search_as_you_type

This field type creates several sub-fields for you: the main field plus ._2gram, ._3gram and ._index_prefix. You query them together with a multi_match of type bool_prefix:

JSON
PUT /articles
{
  "mappings": {
    "properties": {
      "title": { "type": "search_as_you_type" }
    }
  }
}

GET /articles/_search
{
  "query": {
    "multi_match": {
      "query": "quick bro",
      "type": "bool_prefix",
      "fields": ["title", "title._2gram", "title._3gram"]
    }
  }
}

All words but the last must match as whole terms, and the last is matched as a prefix. The query quick bro finds "The quick brown fox" because quick matches exactly and bro is a prefix of brown. This handles words in the middle of a title, which the completion suggester cannot do.

Edge n-grams

An edge n-gram is a prefix of a token. At index time, an analyzer can split brown into b, br, bro, brow, brown. Then a normal match query for bro finds it, because that prefix is simply a term in the index. The analysis chain looks like this (see Custom Analyzers, Synonyms and Stemming for how to build your own):

JSON
PUT /articles
{
  "settings": {
    "analysis": {
      "filter": {
        "edge_filter": { "type": "edge_ngram", "min_gram": 2, "max_gram": 15 }
      },
      "analyzer": {
        "autocomplete_index": {
          "tokenizer": "standard",
          "filter": ["lowercase", "edge_filter"]
        },
        "autocomplete_search": {
          "tokenizer": "standard",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "autocomplete_index",
        "search_analyzer": "autocomplete_search"
      }
    }
  }
}

The important detail is search_analyzer. The query text bro must not itself be expanded into b, br, bro, or it would match everything starting with b. Index time expands, search time does not. The cost of edge n-grams is index size: every token becomes up to max_gram - min_gram + 1 terms. Keep min_gram at 2 or 3 and max_gram modest.

Try it: edge n-gram autocomplete

The stand-in builds an inverted index of edge n-grams and answers prefix queries by looking up terms, then ranks by a popularity weight. Notice how the index-time analyzer and search-time analyzer differ.

JS Edge n-gram autocomplete

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that b returns nothing: a one-letter prefix is shorter than min_gram, so no such term exists in the index. That is a deliberate size trade-off. The query quick bro works because each token is looked up on its own, so word order and position are free.

Did you mean: the suggesters

Suggesters also work on spelling. They look at the terms in your index, not at a dictionary, so they suggest words that really exist in your data.

Term suggester

The term suggester corrects each word independently:

JSON
POST /articles/_search
{
  "suggest": {
    "fix_spelling": {
      "text": "elasticsaerch tutorail",
      "term": {
        "field": "title",
        "suggest_mode": "missing"
      }
    }
  }
}

The response lists options per input word, each with a score and freq (how many documents contain the candidate). suggest_mode controls when to suggest: missing only for words not in the index, popular for words with a more frequent alternative, always for everything.

Phrase suggester

The phrase suggester works on the whole phrase and uses an n-gram language model of your own text to choose corrections that fit together:

JSON
POST /articles/_search
{
  "suggest": {
    "did_you_mean": {
      "text": "quik brwn fox",
      "phrase": {
        "field": "title.trigram",
        "gram_size": 3,
        "max_errors": 2,
        "highlight": { "pre_tag": "<em>", "post_tag": "</em>" },
        "direct_generator": [
          { "field": "title.trigram", "suggest_mode": "always" }
        ]
      }
    }
  }
}

It needs a shingle-based sub-field such as title.trigram (an analyzer with a shingle filter). Setup is heavier than the term suggester, but results are far better for multi-word queries because "brwn" is corrected knowing that "fox" follows.

A cheaper alternative is to run your main query with fuzziness: AUTO (see Full-Text Queries) and, when there are zero or very few hits, show a "Did you mean" line from the phrase suggester. Use the suggester output as a link: do not silently replace what the user typed.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which tool answers autocomplete in a fraction of a millisecond from an in-memory structure?

    The completion field type stores prefixes in an FST kept in memory, which is why it is so fast. It matches from the start of each input.
  2. 2. Why does an edge n-gram setup need a different search_analyzer?

    At index time brown becomes br, bro, โ€ฆ At search time the query bro should stay as one term and look up exactly that prefix term.
  3. 3. You highlight a field that may contain user-written HTML. What should you set?

    Highlighting does not escape content by default. The html encoder escapes the text so only your highlight tags remain as markup.
  4. 4. What is the main difference between the term and phrase suggesters?

    The phrase suggester scores whole candidate phrases with a language model built from your data, so corrections are more coherent.

Summary

  • highlight wraps matched words in tags; use the html encoder for untrusted content.
  • The completion suggester is the fastest autocomplete but only matches prefixes of listed inputs.
  • search_as_you_type with a bool_prefix multi_match matches partial last words anywhere in a title.
  • Edge n-grams build prefixes at index time; use a plain search_analyzer and keep gram sizes modest.
  • The term and phrase suggesters offer "did you mean" from your own index terms.

Next: Custom Analyzers, Synonyms and Stemming.