Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 43 · Analyzers in Practice

Multilingual Search, Accents and N-Grams

Handle accents with asciifolding and ICU, index several languages with per-language fields, search CJK text, and choose between n-grams and edge n-grams.

A search box on a global site meets text in many languages. A French user types cafe and expects "café". A German visitor searches Strasse and the page says "Straße". Chinese and Japanese text has no spaces between words, so the standard splitting rules do not work as people expect. And everyone wants to find a product by typing a piece of its name, which needs partial matching.

This lesson covers the tools for each problem: accent folding, ICU text handling, per-language fields, CJK analysis, and n-grams for partial matching, including the cost of each. The Elasticsearch requests are read-only. A JavaScript stand-in shows folding, language fields and n-gram matching on small data. It is a simplification of what the analysis chain does.

Four multilingual problems

Accents cafe should find café.
Many languages One analyzer cannot fit them all.
CJK No spaces between words.
Partial words Find a name from a fragment.
1 / 2

Accents and normalization

Accents are the easy case. The asciifolding token filter converts characters outside basic ASCII to their ASCII equivalent when one exists: é becomes e, ü becomes u, ß becomes ss. Put it after lowercase:

JSON
PUT /menu
{
  "settings": {
    "analysis": {
      "analyzer": {
        "folded": {
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "dish": { "type": "text", "analyzer": "folded" }
    }
  }
}

Now both café and cafe produce the token cafe, in documents and in queries. You can set "preserve_original": true on the filter to also keep the accented token, which lets an exact accented query score a little better than an unaccented one.

Folding is a blunt tool: in some languages a letter with a diacritic is a different letter, and folding merges words that have different meanings. In Swedish, å, ä and ö are separate letters, and in Spanish año (year) and ano mean different things. Decide per language, ideally with native speakers' feedback.

The ICU plugin

For serious international text, the analysis-icu plugin (a separate plugin you install on every node; it is available on Elastic Cloud) provides tools based on the Unicode standard:

  • icu_tokenizer: Unicode-aware word splitting, including dictionary-based splitting for Thai, Lao, Khmer and Chinese or Japanese text.
  • icu_normalizer and icu_folding: Unicode normalization and a much more complete folding than asciifolding.
  • icu_collation_keyword: language-specific sorting, so that ä sorts where a German user expects.
JSON
{
  "analysis": {
    "analyzer": {
      "intl": {
        "tokenizer": "icu_tokenizer",
        "filter": ["icu_normalizer", "icu_folding"]
      }
    }
  }
}

If you cannot install plugins, the built-in standard tokenizer plus lowercase and asciifolding already handles most European text.

One field per language

One text field has one analyzer. If your documents come in several languages, the usual design is to store each translation in its own field and use the analyzer made for that language (see Custom Analyzers, Synonyms and Stemming):

JSON
PUT /articles
{
  "mappings": {
    "properties": {
      "language": { "type": "keyword" },
      "title": {
        "type": "text",
        "fields": {
          "en": { "type": "text", "analyzer": "english" },
          "de": { "type": "text", "analyzer": "german" },
          "fr": { "type": "text", "analyzer": "french" }
        }
      }
    }
  }
}

At index time, fill only the sub-field that matches the document's language, or index the same text into the one the application knows is right. At search time you have two options:

  1. Know the user's language (from the page, the account or the browser) and query the matching field, perhaps with a filter on the language keyword field.
  2. Search every language field with multi_match of type most_fields when the language is not known. This costs more and can produce odd matches, since a stemmed English word may collide with a German word.

An alternative layout is one index per language. That gives each language clean statistics for scoring, since the inverse document frequency (see Relevance Scoring with BM25) is computed only from documents in that language. It also multiplies the number of shards, so use it when the amount of data per language is large.

Never rely on automatic language detection at query time for short queries. Two or three words are too short for reliable detection.

CJK text

Chinese, Japanese and Korean do not separate words with spaces. The standard tokenizer splits each Chinese character into its own token, so 搜索引擎 ("search engine") becomes four single-character tokens. That finds many false positives.

You have three approaches:

  • The cjk analyzer turns the text into overlapping bigrams (pairs of neighboring characters): 搜索引擎 becomes 搜索, 索引, 引擎. It is simple, needs no dictionary and has surprisingly good recall, at the price of a larger index and some false matches.
  • Dictionary-based plugins such as analysis-smartcn for Chinese, analysis-kuromoji for Japanese and analysis-nori for Korean segment real words with morphology. They give better precision and also handle conjugations in Japanese and Korean.
  • ICU tokenizer with dictionary-based segmentation as a middle path when you cannot add language-specific plugins.

Test with real queries from native speakers before you pick one. Whichever you choose, set the same analyzer for index and search time.

N-grams for partial matching

An n-gram splits a token into overlapping pieces of N characters. The word search with 3-grams gives sea, ear, arc, rch. Unlike an edge n-gram, which only keeps prefixes (sea, sear, searc...), a plain n-gram keeps pieces from anywhere in the word. That lets a query find a word from the middle: arc finds search.

JSON
PUT /parts
{
  "settings": {
    "index": { "max_ngram_diff": 2 },
    "analysis": {
      "tokenizer": {
        "trigrams": { "type": "ngram", "min_gram": 3, "max_gram": 4, "token_chars": ["letter", "digit"] }
      },
      "analyzer": {
        "partial": { "tokenizer": "trigrams", "filter": ["lowercase"] },
        "partial_search": { "tokenizer": "standard", "filter": ["lowercase"] }
      }
    }
  },
  "mappings": {
    "properties": {
      "sku": { "type": "text", "analyzer": "partial", "search_analyzer": "partial_search" }
    }
  }
}

Two settings need attention. max_ngram_diff caps the difference between max_gram and min_gram; the default is 1, so a wider range requires raising it explicitly. And, just as with edge n-grams in Highlighting, Autocomplete and "Did You Mean", use a plain search_analyzer, so the query is not broken into n-grams too.

Comparing the options

Technique Matches Index size Typical use
Normal analysis Whole words Smallest Prose, titles
Edge n-gram Word starts Medium Autocomplete
N-gram Any fragment Largest Part numbers, product codes, "contains"
wildcard field type Any pattern, via an optimized structure Moderate Logs, identifiers with *foo* queries

For the last one, the wildcard field type is built for fast * and ? queries on strings such as log lines and codes without paying the full n-gram cost, and is worth considering before you build n-gram analyzers for "contains" search. Plain wildcard queries with a leading * on a normal keyword field must check many terms and are slow (see Query Performance and Tuning).

Try it: folding, language fields and n-grams

The stand-in demonstrates three ideas on tiny data: folding with Unicode normalization (normalize("NFD") and removing combining marks, a stand-in for asciifolding), a per-language stop word choice, and an n-gram index compared with an edge n-gram index.

JS Accents, languages and n-grams

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The plain n-gram index finds harg inside both "chargeur" and "recharge", which the edge n-gram index cannot, but it stores more terms and returns matches the user may not expect. That is the core trade-off: the more flexible the partial matching, the larger the index and the noisier the results.

Try it: CJK bigrams

This second stand-in shows why single characters are a poor token for Chinese and how bigrams change the match. It uses a tiny made-up corpus and makes no claim about a real segmenter.

JS Characters versus bigrams

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With single characters, document 3 matches two of four tokens and document 2 matches two as well, so a loose or query returns both. With bigrams, only document 1 contains all of 搜索, 索引 and 引擎, so requiring all tokens (or a high minimum_should_match) ranks it clearly first. Documents 2 and 3 each share a single bigram.

Practical advice

  • Start with lowercase plus asciifolding for European text; add ICU when you need Unicode-aware behavior.
  • Prefer one field per language and pick the field from known context, not from guessing.
  • Choose a CJK strategy with native-speaker queries as test data.
  • Use edge n-grams for autocomplete, n-grams or the wildcard field type only when you need "contains" search, and measure the index size.
  • Keep index and search analyzers compatible, apart from deliberate differences such as n-grams.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. What does the asciifolding token filter do?

    It maps accented and special letters to ASCII where an equivalent exists, so café and cafe produce the same token.
  2. 2. How should you usually store documents in several languages for stemming and stop words to work well?

    A field has one analyzer, so each language gets its own field (or index) with a matching analyzer.
  3. 3. Why does the standard tokenizer work poorly for Chinese text?

    Without spaces, the standard tokenizer emits one token per character, which causes many false matches. Bigrams or dictionary-based analyzers do better.
  4. 4. A query for harg should find recharge. Which technique supports that?

    Edge n-grams only keep prefixes of each token. Plain n-grams keep pieces from anywhere, so harg is among the stored terms.
  5. 5. What does index.max_ngram_diff control?

    The default difference is 1. Wider ranges must be allowed explicitly, since they produce many more terms.

Summary

  • asciifolding (or ICU folding) makes accented and unaccented spellings match; apply it with care for languages where accents change meaning.
  • Use one field or index per language, with the analyzer for that language, and choose the field from known context.
  • For CJK text use bigrams (cjk), a dictionary plugin, or the ICU tokenizer, and test with native queries.
  • Edge n-grams match prefixes; plain n-grams match fragments but make the index larger and noisier.
  • Consider the wildcard field type before building heavy n-gram analyzers for "contains" search.

Next: Aggregations: Metrics and Buckets.