Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 43 ยท Beyond Keywords

Percolator and Alerting Patterns

Use the Elasticsearch percolate query for reverse search: store queries, match incoming documents against them, and build saved searches and alert-on-match pipelines.

A normal search stores documents and runs a query against them. Sometimes the situation is the other way round: you have thousands of saved queries, and a new document arrives. Which of those queries does it match? A news site wants to tell every reader whose saved search is "electric AND bicycle" about a new article. A monitoring system wants to know which alert rules a log line triggers.

You could loop over the queries and run each one, but that is slow. Elasticsearch has a feature built for it: the percolator. It is called "reverse search" because queries are the stored data and documents are the questions. The cluster cannot run on this site, so the requests are read-only and the runnable examples are JavaScript stand-ins.

Storing queries

A percolator index has a field of type percolator that holds a query. Its mapping must also define the fields those queries refer to, because the queries are parsed when stored:

HTTP
PUT /saved-searches
{
  "mappings": {
    "properties": {
      "query":   { "type": "percolator" },
      "title":   { "type": "text" },
      "level":   { "type": "keyword" },
      "price":   { "type": "float" },
      "user_id": { "type": "keyword" }
    }
  }
}

Here title, level and price describe the documents you will match against. user_id is an ordinary field on the stored query document, so you can filter which saved searches to run. Now store searches as documents:

HTTP
PUT /saved-searches/_doc/1
{
  "user_id": "ada",
  "query": { "match": { "title": "electric bicycle" } }
}

PUT /saved-searches/_doc/2
{
  "user_id": "bo",
  "query": {
    "bool": {
      "must":   { "match": { "title": "bicycle" } },
      "filter": { "range": { "price": { "lt": 500 } } }
    }
  }
}

Any query type is allowed inside, with a few exceptions such as queries that fetch other documents at run time.

Send the incoming document inside a percolate query. The response lists the stored queries that match it:

HTTP
GET /saved-searches/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": { "title": "Folding electric bicycle on sale", "price": 450 }
    }
  }
}

Both saved searches above match this document, so both come back as hits, and each hit says which query it is. You can percolate several documents at once with documents, and then each hit includes _percolator_document_slot so you know which document matched. To narrow the work, add a filter on the stored query documents, for example { "term": { "user_id": "ada" } } next to the percolate clause inside a bool.

Search versus percolate

Normal search Stored documents, one query comes in.
Percolate Stored queries, one document comes in.
Result The list of queries that match.

Same query language, opposite direction.

1 / 2

How it stays fast

You might expect that every stored query runs against the new document. Elasticsearch avoids most of that work. When a query is stored, it extracts the terms any matching document must contain, and keeps them in the index. At percolation time, the incoming document is first compared with those extracted terms to find candidate queries, and only the candidates are executed against a small in-memory index that holds the document. That is why percolation scales to many thousands of saved queries.

Some things to remember:

  • Queries are parsed when stored, so the mapping has to contain the fields, and a mapping change affects how old queries behave. Reindex the percolator index after changing analyzers.
  • A stored query uses the analyzers of the percolator index's fields, so the saved search and the incoming document are analyzed in the same way.
  • Highlighting works: add highlight to show which words of the document matched a saved query.

Alert-on-match patterns

A percolator answers "which rules match this event". It does not push anything by itself. The usual pipeline looks like this:

  1. Your ingest service receives a document (a log line, an article, a product update).
  2. Before or just after indexing it, the service sends a percolate request with that document.
  3. For each matching saved search, the service creates a notification: an email, a chat message, a webhook, a row in an inbox table.
  4. The service avoids duplicates by recording which query and document pairs were already sent.

Cost and design notes:

  • Percolate one batch at a time. Sending 100 documents in one request is much cheaper than 100 requests.
  • Keep saved queries simple. Unusually broad queries such as wildcards or very common words make many candidates and slow everything down.
  • Separate the percolator index from your main index. It has its own mapping, size and refresh needs.
  • Use the built-in alerting when you can. Kibana alerting and Watcher run searches on a schedule over stored data, which is a better fit for "tell me when error counts rise" than percolation. The percolator fits "tell me about each new item that matches" in near real time, before or as the item is stored.

The stand-in below keeps saved queries as small functions, and percolates documents against them. It also shows the candidate step: queries are first filtered by their required terms, and only then run in full.

JS Reverse search: queries stored, documents arrive

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The real percolator works on analyzed terms and extracts them from arbitrary query structures, which is far more general. The stand-in only shows the two ideas: a cheap candidate filter first, and the full check second.

An alert pipeline you can run

This second stand-in turns matches into notifications and avoids sending the same alert twice, as a real consumer of percolate results should.

JS From matches to notifications without duplicates

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A real service would store the sent keys in a database and expire them. The important habit is that the consumer is idempotent: queues deliver at least once (see Delivery Semantics), so a retry must not cause a second email.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the percolate query return?

    Percolation is reverse search: queries are stored, and the document you send is matched against them.
  2. 2. Why must the mapping of a percolator index define the fields used in stored queries?

    A stored query needs field types and analyzers to be parsed, and the incoming document must be analyzed the same way.
  3. 3. Which task is a better fit for Kibana alerting or Watcher than for the percolator?

    Threshold alerts are scheduled searches over stored data. The percolator is for per-document matching against many saved queries.
  4. 4. What should a consumer of percolate results do about duplicate deliveries?

    The pipeline around the percolator can retry or deliver events more than once, so notifications must be idempotent.

Summary

  • The percolator stores queries in a percolator field and matches incoming documents against them: reverse search.
  • The mapping must define the fields used by the saved queries; analyzers are shared between queries and documents.
  • Elasticsearch extracts required terms from stored queries to pick candidates quickly, then runs only those.
  • Percolation does not notify anyone. Your service turns matches into alerts, in batches, and must avoid duplicates.
  • Use scheduled alerting for threshold rules, and the percolator for "this new item matches a saved search".

Next: return to the main search pipeline with Vector and Semantic Search, another way to find what is related.