Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 43 ยท How Search Works

Analysis, Tokenizers and Analyzers

Understand how Elasticsearch turns text into terms: character filters, tokenizers, token filters, the standard, whitespace and keyword analyzers, and _analyze.

When you index the sentence "The QUICK brown foxes!", Elasticsearch does not store that string as one searchable value. It first runs the text through analysis, which produces a list of terms, and those terms go into the inverted index (The Inverted Index). When you search, your query text is analyzed too. A match happens when query terms and document terms are equal.

That is why analysis is the single most important concept for good search. If the document is indexed as foxes and your search becomes fox, they do not match, unless analysis made them the same. This lesson shows the three stages of an analyzer, the built-in analyzers you will meet first, and the _analyze API for testing them. Real requests are read-only; the JavaScript pipeline later is a stand-in you can change step by step.

The analysis chain

An analyzer is a small pipeline of exactly three kinds of parts, always in this order:

  1. Character filters (zero or more) clean the raw text before it is split. Example: html_strip removes tags, so <b>Fast</b> becomes Fast. Another one maps characters, for example & to and.
  2. A tokenizer (exactly one) cuts the text into tokens, roughly words. It also records each token's position and character offsets.
  3. Token filters (zero or more) change, remove or add tokens. Examples: lowercase, stop (removes words like "the"), stemmer (cuts "running" to "run"), synonym, asciifolding (turns "cafรฉ" into "cafe").
flowchart LR
    A["Raw text<br/>'The QUICK foxes!'"] --> B["Character filters"]
    B --> C["Tokenizer"]
    C --> D["Token filters"]
    D --> E["Terms<br/>quick, fox..."]

Text flows through character filters, one tokenizer and token filters, and the resulting terms are what the index stores.

The same chain, or a deliberately different one, runs again on the query text. If the two chains disagree, searches silently miss documents; this mistake appears in Custom Analyzers, Synonyms and Stemming.

Built-in analyzers

Elasticsearch ships with several ready-made analyzers. Three matter first.

standard is the default for text fields. It uses the standard tokenizer, which splits on word boundaries following Unicode rules (so it handles many languages reasonably), then applies the lowercase token filter. It removes most punctuation. In current versions it does not remove stop words by default.

whitespace splits on spaces only. It does no lowercasing and keeps punctuation attached to words, so Hello, and hello are different terms. It suits data where case and symbols matter, such as some codes.

keyword does nothing: the whole input becomes one token. It is how exact-value fields work (keyword fields use a similar idea, see Mappings and Field Types).

Other built-ins include simple (splits on anything that is not a letter, then lowercases), stop (simple plus stop words), pattern, and one analyzer per language such as english, german and french, which add language-specific stop words and stemming.

Three stages of an analyzer

Character filters Clean the raw text first.
Tokenizer Cut text into tokens. Exactly one.
Token filters Lowercase, stop, stem, synonyms.

Order matters: filters run left to right.

1 / 2

The _analyze API

You never have to guess what an analyzer does, because the _analyze API shows the tokens it produces:

JSON
GET /_analyze
{
  "analyzer": "standard",
  "text": "The QUICK brown foxes!"
}

The response lists each token with its position and offsets:

JSON
{
  "tokens": [
    { "token": "the",   "start_offset": 0,  "end_offset": 3,  "type": "<ALPHANUM>", "position": 0 },
    { "token": "quick", "start_offset": 4,  "end_offset": 9,  "type": "<ALPHANUM>", "position": 1 },
    { "token": "brown", "start_offset": 10, "end_offset": 15, "type": "<ALPHANUM>", "position": 2 },
    { "token": "foxes", "start_offset": 16, "end_offset": 21, "type": "<ALPHANUM>", "position": 3 }
  ]
}

You can also test your own combination without creating an index, by naming the parts:

JSON
GET /_analyze
{
  "char_filter": ["html_strip"],
  "tokenizer": "standard",
  "filter": ["lowercase", "stop"],
  "text": "<p>The Quick Fox</p>"
}

And with an index that has a field mapped to a particular analyzer, GET /my-index/_analyze with "field": "title" shows exactly what that field does. Whenever a search surprises you, run _analyze on both the document text and the query text first.

When the tokens do not match

Two common surprises:

  • A match query on a text field for Fox finds fox documents, because both sides are lowercased.
  • A term query for Fox on the same field finds nothing: a term query is not analyzed, and the index contains only lowercase fox. The reverse also happens: a term query for Quick Brown on a keyword field only matches the exact string.

Rule of thumb: use match for analyzed text and term for exact values. Queries are covered in Query DSL Basics.

An analysis pipeline you can change

The stand-in below implements the three stages with plain JavaScript. The tokenizer is a close but not exact copy of the standard tokenizer (the real one follows the full Unicode text segmentation rules), and the stemmer is a deliberately tiny toy; real stemmers like the Porter algorithm are much more careful. Edit the configurations at the bottom to see how each stage changes the output.

JS Build an analyzer from parts

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Compare the lines. keyword returns the entire string as one token, markup included. whitespace keeps <p>The, QUICK and punctuation. standard splits on word boundaries and lowercases, but it treats the markup as text (p appears twice) and keeps the. Adding html_strip and stop removes the tags and the, and the toy stemmer turns foxes, running and jumping into fox, runn and jump: crude, but enough to show why a query for jumps could match a document containing jumping. (Note runn: poor stemmers make poor terms, which is why you should use the language stemmers that ship with Elasticsearch.)

Matching through analysis

To see the whole effect, let a query and a document go through the same chain and compare terms. This second stand-in shows a hit appear only after analysis.

JS Why analysis makes a match

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Without stemming, fox misses foxes and forests misses forest; capital letters are fine because of lowercasing. With stemming both match. The query the analyzes to no terms at all (it was a stop word), so nothing can match it: another common surprise when stop words are removed.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the correct order of the parts of an analyzer?

    Character filters clean the raw text, the single tokenizer cuts tokens, and token filters then modify the token list.
  2. 2. A document field uses the standard analyzer. A term query searches for "Fox" but the document text was "Fox". Why is there no hit?

    The standard analyzer lowercased the term at index time. A term query looks for the exact string you give it, so Fox does not equal fox.
  3. 3. What does the keyword analyzer produce for the text "New York City"?

    The keyword analyzer treats the whole input as a single token, which is useful for exact values.
  4. 4. How can you check what terms a field actually indexes?

    _source only returns your original JSON. _analyze shows the tokens an analyzer produces.

Summary

  • Analysis turns text into terms using character filters, one tokenizer and token filters, in that order.
  • The standard analyzer (the default) splits on word boundaries and lowercases; whitespace splits on spaces only; keyword makes one token.
  • Documents and match queries are analyzed with matching rules; term queries are not analyzed.
  • _analyze shows exactly which tokens a given analyzer or field produces: use it whenever matches surprise you.
  • Stemming and stop words trade precision for recall and can make a query match nothing.
  • The JavaScript pipeline is a simplified model, not Elasticsearch's exact tokenizer or stemmers.

Next: Relevance Scoring with BM25.