Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 43 ยท Working from Code

Ingest Pipelines and Processors

Transform documents inside Elasticsearch before they are indexed: processors such as grok, date, rename, set and script, the simulate API, default_pipeline and failure handling.

Data rarely arrives in the shape you want to search. A log line is one long string, a timestamp is text, a field has the wrong name, an IP address has no location yet. You could fix all of that in your application before calling the index API, but then every producer needs the same code. An ingest pipeline moves the fixing into Elasticsearch itself: documents pass through a list of small steps, called processors, on an ingest node, and only the result is indexed.

This lesson shows how to define a pipeline, which processors you will use most, how to test one with the simulate API, and how to handle failures. The cluster cannot run on this site, so the REST requests are read-only. The runnable examples are plain JavaScript stand-ins that imitate a few processors so you can see how a document changes step by step. They are not Elasticsearch's real implementation.

What a pipeline does

Raw document A log line or messy JSON arrives.
Processors Each step edits the document.
Index Clean, typed fields are stored.

Parse once at ingest and every query stays cheap.

1 / 2

Defining a pipeline

A pipeline is a named, ordered list of processors stored in the cluster state:

HTTP
PUT /_ingest/pipeline/web-logs
{
  "description": "Parse access log lines",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{IPORHOST:client_ip} %{WORD:method} %{URIPATH:path} %{NUMBER:status:int} %{NUMBER:took_ms:float}"]
      }
    },
    { "date": { "field": "timestamp", "formats": ["ISO8601"], "target_field": "@timestamp" } },
    { "rename": { "field": "client_ip", "target_field": "client.ip" } },
    { "set": { "field": "event.ingested", "value": "{{{_ingest.timestamp}}}" } },
    { "remove": { "field": ["timestamp", "message"], "ignore_missing": true } }
  ]
}

Processors run top to bottom, and each one sees the output of the previous one. Names with dots like client.ip create nested objects. The {{{...}}} syntax is a template (Mustache) that can read other fields or the special _ingest metadata.

You use a pipeline by naming it in the request:

HTTP
PUT /web-logs-2026.11/_doc/1?pipeline=web-logs
{ "timestamp": "2026-11-06T10:15:00Z", "message": "203.0.113.7 GET /checkout 502 1840.5" }

POST /_bulk?pipeline=web-logs

The pipeline parameter works on single-document and bulk requests alike, as discussed in using Elasticsearch from code.

The processors you will use most

Processor What it does
grok Matches a regular-expression-based pattern and extracts named fields from text.
dissect Splits text on fixed delimiters. Faster and simpler than grok when the format is rigid.
date Parses a date string with the formats you list and writes @timestamp (or another field).
rename, remove, set Move a field, delete fields, or assign a value (static or from a template).
convert Changes a field's type, for example "42" to an integer.
lowercase, trim, split, join Small text and array clean-ups.
script Runs a Painless script for logic the other processors cannot express.
geoip Looks up an IP address in a bundled geo database and adds country, city and coordinates.
enrich Joins the document with data from another index through a prepared enrich policy.
pipeline Calls another pipeline, so you can build small reusable pieces.
attachment or inference Extract text from binary files or run a model (availability depends on version and licence).

Most processors accept ignore_missing: true (do nothing if the field is absent) and if (a Painless condition, run only when true). Use if to branch:

HTTP
{ "set": { "field": "event.severity", "value": "high", "if": "ctx.status != null && ctx.status >= 500" } }

Inside scripts and conditions, ctx is the document being processed.

Enriching with geoip and enrich

HTTP
{ "geoip": { "field": "client.ip", "target_field": "client.geo", "ignore_missing": true } }

geoip uses a database shipped with or downloaded by the cluster. The enrich processor is for your own lookup data, for example mapping a product id to its category from a reference index. You create an enrich policy, execute it to build a hidden lookup index, and then reference the policy by name in the processor. Enrichment is a copy at ingest time: if the reference data changes later, old documents keep the old values.

Testing with the simulate API

Do not try pipelines out by indexing real data. The simulate API runs sample documents through a pipeline and returns the result without storing anything:

HTTP
POST /_ingest/pipeline/web-logs/_simulate
{
  "docs": [
    { "_source": { "timestamp": "2026-11-06T10:15:00Z", "message": "203.0.113.7 GET /checkout 502 1840.5" } },
    { "_source": { "timestamp": "not a date", "message": "bad line" } }
  ]
}

Add ?verbose=true to see the document after every single processor, which is the fastest way to find which step breaks. You can also send an inline pipeline definition instead of a name while you are still drafting.

A pipeline stand-in you can run

The stand-in below implements split-style parsing, convert, rename, set with a condition and remove, and prints the document after each step, like verbose=true.

JS Run a document through processors

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The real grok processor uses a library of named regular expressions (IPORHOST, NUMBER) instead of \S+, and the real processors run inside the cluster, but the flow is identical: each step reads and edits one shared document.

When a processor fails

By default a failing processor rejects the whole document and the client sees an error. You have two tools:

  • ignore_failure: true on one processor skips it silently. Use it only for optional enrichment.
  • on_failure runs a list of processors instead, with the error details in _ingest.on_failure_message. A common choice is to tag the document and keep it, or to reroute it to a dead-letter index.
HTTP
PUT /_ingest/pipeline/web-logs-safe
{
  "processors": [
    { "grok": { "field": "message", "patterns": ["%{IPORHOST:client_ip} %{WORD:method}"] } }
  ],
  "on_failure": [
    { "set": { "field": "error.message", "value": "{{{_ingest.on_failure_message}}}" } },
    { "set": { "field": "_index", "value": "failed-{{{_index}}}" } }
  ]
}

Changing _index in a failure handler sends the broken document to a different index, so you can inspect and replay it instead of losing it.

JS Keep failed documents in a failure index

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Default and final pipelines

Instead of asking every client to add ?pipeline=, set it on the index so every write goes through it:

HTTP
PUT /web-logs-2026.11
{
  "settings": {
    "index.default_pipeline": "web-logs",
    "index.final_pipeline": "add-ingested-time"
  }
}

The default pipeline runs when the request names none. The final pipeline always runs last, even if the request named a different pipeline, which makes it good for rules that must never be skipped, such as scrubbing sensitive fields. Put these settings in an index template (dynamic mapping and templates) so new daily indexes inherit them.

Indexing attachments and binary files

PDFs, Word documents and slide decks are binary, so Elasticsearch cannot index them directly. The attachment processor, from the ingest-attachment plugin, extracts text and metadata with Apache Tika. The plugin must be installed on every node that runs ingest, then the nodes restarted:

Terminal
bin/elasticsearch-plugin install ingest-attachment

You send the file as a Base64 string in a field, and the pipeline replaces it with extracted data:

JSON
PUT /_ingest/pipeline/docs-attachment
{
  "description": "Extract text from uploaded files",
  "processors": [
    {
      "attachment": {
        "field": "data",
        "target_field": "attachment",
        "indexed_chars": 200000,
        "remove_binary": true
      }
    }
  ]
}

PUT /library/_doc/1?pipeline=docs-attachment
{
  "filename": "handbook.pdf",
  "data": "JVBERi0xLjQKJ..."
}

The result holds attachment.content (the text), plus metadata such as content_type, content_length, title, author, date, keywords and language when the file provides them. Useful options:

  • indexed_chars limits how much text is extracted (the default is 100,000 characters; -1 means all). Long documents beyond the limit are silently cut, so raise it deliberately.
  • properties chooses which of the fields above to keep.
  • remove_binary: true drops the Base64 text from the stored document; otherwise the source keeps a copy of the whole file, which wastes disk.
  • ignore_missing lets documents without the field pass through.

Search and filter the extracted fields like any other:

JSON
GET /library/_search
{
  "query": {
    "bool": {
      "must":   { "match": { "attachment.content": "vacation policy" } },
      "filter": { "term":  { "attachment.content_type": "application/pdf" } }
    }
  }
}

To index several files in one document, put them in an array of objects and use a foreach processor that runs attachment over each element. For large files, or heavy volumes, run Tika or a crawler outside the cluster and index only the extracted text, because extraction uses a lot of CPU and memory on the ingest nodes and Base64 inflates the request by about a third. Like the rest of this lesson, these requests are read-only here; the site cannot run Elasticsearch or its plugins.

Practical advice

  • Keep pipelines small and name them for their purpose. Compose them with the pipeline processor.
  • Always test with _simulate and include bad input, not just the happy path.
  • Ingest work uses CPU on ingest nodes. Heavy parsing at high volume is a reason to use dedicated ingest nodes or to parse in Logstash or a Beat.
  • Pipelines change only documents indexed after the change. Existing documents are not touched unless you run _update_by_query or _reindex with a pipeline.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the simulate API do?

    Simulate is a dry run. With verbose=true it also shows the document after each processor.
  2. 2. Which setting makes sure a pipeline runs even if the request named a different pipeline?

    default_pipeline only applies when the request names none. final_pipeline always runs last.
  3. 3. A processor fails and you want to keep the document in another index for inspection. What do you use?

    on_failure can record the error message and reroute the document by setting _index.
  4. 4. After you edit a pipeline, what happens to documents that are already indexed?

    Pipelines run at index time only, so old documents keep their old shape.

Summary

  • An ingest pipeline is an ordered list of processors that edits documents before they are indexed.
  • Use grok or dissect to split text, date for timestamps, convert, rename, set, remove for shaping, script for custom logic, and geoip or enrich for extra data.
  • Test with POST /_ingest/pipeline/<name>/_simulate, adding verbose=true to see each step.
  • Handle failures with ignore_failure or an on_failure block, and reroute bad documents instead of losing them.
  • Set index.default_pipeline and index.final_pipeline in templates so every writer gets the same treatment.

Next: collect the data in the first place with Logstash and Beats.