Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 43 ยท Analyzers in Practice

Custom Analyzers, Synonyms and Stemming

Build Elasticsearch custom analyzers in index settings: stemmers, stop words, synonym sets, language analyzers and index-time vs search-time analysis.

The built-in standard analyzer is a good start, but real search needs more. Shoppers type sneakers and expect sneaker to match. Someone searching tv should find "television". Common words like "the" add noise and size to the index. You get all of this by building your own analyzer from the same three kinds of parts you met in Analysis, Tokenizers and Analyzers: character filters, one tokenizer and token filters.

This lesson shows how to declare a custom analyzer in index settings, how stemming, stop words and synonyms work, how language analyzers bundle those choices, and why index-time and search-time analysis may differ. The Elasticsearch requests are read-only. The JavaScript stand-in is a toy pipeline with a deliberately simple stemmer, so you see the effects without a server.

Pieces of an analyzer

Char filters Clean raw text first.
Tokenizer Split text into tokens. Exactly one.
Token filters Lowercase, stem, drop, add synonyms.

Order matters. Filters run top to bottom.

1 / 2

Declaring a custom analyzer

Analyzers live in the settings.analysis section of an index. You define reusable filters first, then an analyzer that lists them in order:

JSON
PUT /shop
{
  "settings": {
    "analysis": {
      "filter": {
        "english_stop": { "type": "stop", "stopwords": "_english_" },
        "english_stemmer": { "type": "stemmer", "language": "english" }
      },
      "analyzer": {
        "product_text": {
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stop", "english_stemmer"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "description": { "type": "text", "analyzer": "product_text" }
    }
  }
}

Test it with the _analyze API before indexing anything:

JSON
POST /shop/_analyze
{
  "analyzer": "product_text",
  "text": "<b>The runners are running</b>"
}

The tags are stripped, The and are are dropped as stop words, and both runners and running reduce to stems. Analysis settings cannot be changed on an open index in a way that alters existing tokens: the data in the index was built with the old rules. To change an analyzer you normally create a new index and reindex (see Using Elasticsearch from Code for zero-downtime reindexing with aliases).

Order matters. lowercase normally goes first so that later filters compare lowercase words. A stop filter after lowercasing needs lowercase stop words; a stemmer before the stop filter might turn a word into something that is no longer in the list.

Stemming

A stemmer reduces words to a root form so related words match. Elasticsearch offers two families:

  • Algorithmic stemmers apply rules: stemmer (with porter2 or english rules) and snowball. They are fast and need no files, but they make mistakes. A stem does not have to be a real word: university and universe may be reduced to the same stem.
  • Dictionary stemmers (hunspell) look words up in a dictionary file. They are more accurate and also more memory-hungry and harder to maintain.

A lighter option is stemmer with light_english (or minimal_english), which only handles plurals and a few endings. For product catalogs, a light stemmer often gives fewer surprises than an aggressive one. The keyword_marker filter before a stemmer lets you protect specific words from stemming.

Stemming increases recall (more matches) and lowers precision (some wrong matches). A common trick is to index text twice: a description field with the stemmed analyzer, and a description.exact sub-field with the plain analyzer, then search both with multi_match of type most_fields and boost the exact one (see Full-Text Queries).

Stop words

Stop words are very common words you drop. The built-in list _english_ removes words such as "the", "a", "and". Since version 5 the default standard analyzer no longer removes stop words, because BM25 scoring (see Relevance Scoring with BM25) already gives common words little weight. Dropping them saves some index space, but it breaks phrase searches like to be or not to be, because every word is a stop word.

If you remove them, do it deliberately and test queries like names, titles and slogans. Many modern setups keep stop words in the index and rely on scoring.

Synonyms

A synonym filter maps words to equivalent words. You can define rules inline or in a managed synonym set:

JSON
PUT /shop
{
  "settings": {
    "analysis": {
      "filter": {
        "product_synonyms": {
          "type": "synonym_graph",
          "synonyms": [
            "tv, television, telly",
            "laptop, notebook",
            "sneakers => trainers"
          ]
        }
      },
      "analyzer": {
        "product_search": {
          "tokenizer": "standard",
          "filter": ["lowercase", "product_synonyms"]
        }
      }
    }
  }
}

There are two rule styles:

  • a, b, c is an equivalence list. Each word expands to all words in the list.
  • a => b is a replacement: the left side is turned into the right side.

Use synonym_graph for search-time analyzers; it understands multi-word synonyms like new york city, nyc. A plain synonym filter is not fully correct for multi-word rules at search time, and a graph filter cannot be used at index time. Newer Elasticsearch versions also manage synonym rules as synonym sets through an API, so you can update them without closing the index; check your version's documentation for the details of that feature.

Index time or search time?

Where you expand synonyms has consequences:

Expand at index time Expand at search time
Changing a rule Requires reindexing Applies to new searches, often just a reload
Index size Larger Same
Multi-word synonyms Unreliable Works with synonym_graph
Scoring Synonym terms add document frequency Cleaner statistics

So the usual advice is: apply synonyms at search time only. You do that by giving the field two analyzers:

JSON
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "product_text",
        "search_analyzer": "product_search"
      }
    }
  }
}

The analyzer runs when documents are indexed; search_analyzer runs on the query text. They must agree on the base tokens (the same tokenizer and lowercase and stemming), otherwise query tokens will never match indexed tokens. Edge n-gram autocomplete, shown in Highlighting, Autocomplete and "Did You Mean", is the classic case where the two deliberately differ.

Language analyzers

For many languages, Elasticsearch ships ready-made analyzers: english, french, german, spanish, russian and more. Each bundles a sensible tokenizer, lowercasing, a stop list and a stemmer for that language. Use one with a single setting:

JSON
{
  "mappings": {
    "properties": {
      "body": { "type": "text", "analyzer": "german" }
    }
  }
}

You can also rebuild a language analyzer as a custom analyzer when you want to change one piece, such as adding your own stop words or a synonym filter. The reference configuration in the documentation shows the exact filter chain per language. One field can only have one analyzer; to support several languages, use several fields (see Multilingual Search, Accents and N-Grams).

Try it: a toy analysis pipeline

The stand-in implements lowercase, stop words, a very small English stemmer and synonyms. The stemmer is a handful of suffix rules, not the real Porter2 algorithm, so it will differ from Elasticsearch on many words. Toggle the pipeline settings to see the effect on matching.

JS A configurable analyzer

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last line shows the classic mistake. When the index and the search disagree on stemming, the query token runner can never match the indexed token runners, and a document that obviously contains the word is not found.

Practical advice

  • Start from a language analyzer, and customize only when you can name a failing query.
  • Use the _analyze API with real titles from your data, both with the field's analyzer and with explain: true to see each step.
  • Keep an "exact" sub-field next to a stemmed field so exact matches can rank first.
  • Keep synonym lists small, reviewed and in version control. Broad synonyms such as phone, device create many false matches.
  • Remember that changing an analyzer means reindexing the stored data.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Where do you declare a custom analyzer?

    Filters, tokenizers and analyzers are defined under settings.analysis, and a field's mapping refers to the analyzer by name.
  2. 2. Why are synonyms usually applied at search time rather than index time?

    Index-time expansion bakes rules into the data, so every rule change needs a reindex. Search-time expansion keeps the index small and flexible.
  3. 3. What goes wrong if the index analyzer stems words but the search analyzer does not?

    Indexed and query tokens must be produced by compatible analysis. Otherwise the query looks for terms that do not exist in the index.
  4. 4. Which synonym rule form replaces one word with another?

    a => b is a replacement rule. A comma-separated list is an equivalence group where each word expands to all of them.
  5. 5. A stemmer reduces universe and university to the same stem. This is an example of:

    Algorithmic stemmers are approximate. Aggressive rules raise recall but can conflate unrelated words.

Summary

  • A custom analyzer is declared in settings.analysis as optional char filters, one tokenizer and ordered token filters.
  • Stemming widens matches; stop words shrink the index but can break phrases; both need testing on real data.
  • Prefer search-time synonym_graph filters, with analyzer and search_analyzer agreeing on the base tokens.
  • Language analyzers bundle a stemmer and stop list per language; one field has one analyzer.
  • Changing analysis rules for existing data means reindexing.

Next: Multilingual Search, Accents and N-Grams.