Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 43 ยท Working from Code

Security, Backups and Operations

Secure and run Elasticsearch: roles, API keys, TLS, snapshots, monitoring, upgrades, a first look at ES|QL, and how to handle user search input safely.

An Elasticsearch cluster usually holds the most searchable copy of your business data: customers, orders, logs. Treat it like a database. It needs authentication, least-privilege access, encrypted connections, backups you have actually restored, and someone watching its health.

This lesson walks through the main controls and the habits that keep a cluster boring, which is what you want. It ends with the part that is entirely in your application's hands: handling user search input safely. The cluster itself cannot run on this site, so the configuration is read-only, and the runnable example is a JavaScript stand-in for input handling.

Never expose port 9200 to the internet without authentication and TLS. Open Elasticsearch clusters with no security have been found and wiped by automated scanners many times. Keep the cluster on a private network and let only your application servers reach it.

Security in layers

Four layers of protection

TLS Encrypt traffic between clients and nodes.
Authentication Users and API keys prove who is calling.
Authorization Roles limit what each caller may do.
Network Only app servers can reach the port.

Remove any one layer and the others carry more risk.

1 / 2

Recent Elasticsearch versions turn security on by default when you start a new node for the first time: the node generates TLS certificates and a password for the built-in elastic superuser, and prints them once. Treat that user like a root account: use it for setup, not for your application.

Users and roles

Access is controlled with role-based access control. A role lists which cluster actions are allowed and which indexes can be read or written, even down to specific fields or documents. Users get roles; applications should get narrow ones.

HTTP
PUT /_security/role/product_search_reader
{
  "cluster": [],
  "indices": [
    {
      "names": ["products*"],
      "privileges": ["read"],
      "field_security": { "grant": ["name", "price", "category"] },
      "query": { "term": { "published": true } }
    }
  ]
}

PUT /_security/user/shop_web
{
  "password": "use-a-long-random-password-here",
  "roles": ["product_search_reader"]
}

This role can only read the products* indexes, can see only three fields, and only documents with published: true. Even if the web app were compromised, the attacker could not read internal fields or delete anything.

API keys

For services, API keys are better than shared passwords: each key has its own name, optional expiry, and a role descriptor that can only be a subset of its creator's permissions. You can revoke one key without touching the others.

HTTP
POST /_security/api_key
{
  "name": "shop-web-search",
  "expiration": "90d",
  "role_descriptors": {
    "search_only": {
      "indices": [ { "names": ["products*"], "privileges": ["read"] } ]
    }
  }
}

The response contains an encoded value, shown only once. Store it in a secret manager or an environment variable, and send it as Authorization: ApiKey <encoded> (the clients take it as an apiKey option, see using Elasticsearch from code). Rotate keys on a schedule, and use separate keys for searching (read) and for indexing (write).

TLS

There are two encrypted channels. HTTP TLS protects traffic between clients and nodes: your app must trust the cluster's certificate authority (CA), either a public one or your own. Transport TLS protects node-to-node traffic inside the cluster, and is required once security is on. Never "fix" certificate errors in production by turning off verification in the client.

Backups: snapshots

Replicas protect against a node failing, not against a mistaken DELETE, a bad reindex, or a corrupted cluster. For that you need snapshots: copies of indexes (and optionally cluster state) stored in a repository, such as a shared filesystem or object storage (S3, Google Cloud Storage, Azure).

HTTP
PUT /_snapshot/nightly_repo
{
  "type": "s3",
  "settings": { "bucket": "my-es-backups", "base_path": "prod" }
}

PUT /_snapshot/nightly_repo/snap-2026-11-06?wait_for_completion=false
{ "indices": "products*,orders*", "include_global_state": false }

POST /_snapshot/nightly_repo/snap-2026-11-06/_restore
{ "indices": "products-v2", "rename_pattern": "(.+)", "rename_replacement": "restored-$1" }

Key facts:

  • Snapshots are incremental: after the first, only new segments are copied, so frequent snapshots are cheap.
  • Do not back up by copying the data directory of a running node. The files are not consistent.
  • Automate with snapshot lifecycle management (SLM): a policy with a schedule and a retention rule.
  • A backup you never restored is only a hope. Restore into a test cluster regularly, using a rename pattern like the one above so it cannot collide with live indexes.

Monitoring

Three checks catch most trouble:

HTTP
GET /_cluster/health
GET /_cat/indices?v&s=store.size:desc
GET /_cat/nodes?v&h=name,heap.percent,cpu,disk.used_percent,node.role
  • Cluster health is green (all shards assigned), yellow (all primaries assigned, some replicas missing) or red (some primary shard has no home, so data is unavailable). See shards and replicas.
  • Disk: at default settings, Elasticsearch stops allocating shards to nodes above a high disk watermark and makes indexes read-only at the flood-stage watermark. A full disk is the most common cause of surprise outages.
  • JVM heap and garbage collection: constant high heap use or long pauses mean too many shards or too heavy queries (see query performance).
  • Rejections: growing numbers of rejected write or search requests mean the cluster is saturated.

Use the monitoring features that fit your setup (Kibana stack monitoring, or an external system such as Prometheus with an exporter) and alert on: cluster health not green for more than a few minutes, disk above 80%, and sustained rejections.

Upgrades

  • Read the breaking changes page for every version you cross, and use the upgrade assistant in Kibana (it lists deprecated settings and indexes that need reindexing).
  • Take a snapshot first.
  • Upgrade one major version at a time (for example 8.x to 9.x, after first reaching the latest 8.x). Indexes created in too old a version may have to be reindexed before a major upgrade.
  • On a multi-node cluster, do a rolling upgrade: upgrade one node at a time, waiting for green between nodes, with upgrades of non-master and data nodes planned in the documented order.
  • Upgrade clients to match, as discussed in the previous lesson.

Troubleshooting a cluster

The basic checks above tell you that something is wrong. This section is about finding out why.

Why is the cluster yellow or red?

Health is derived from shard states. A yellow cluster has every primary assigned but at least one replica unassigned; typical causes are a single-node cluster with number_of_replicas: 1 (a replica can never live on the same node as its primary), a node that left, or disk watermarks blocking allocation. A red cluster has at least one unassigned primary: a node holding the only copy is gone, a shard is corrupt, or a new index cannot be placed. Start by listing the culprits:

HTTP
GET /_cluster/health?level=indices
GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason&s=state

Ask the cluster: allocation explain

When a shard stays UNASSIGNED, do not guess. Ask Elasticsearch to explain its decision:

HTTP
GET /_cluster/allocation/explain
{ "index": "orders-2026.11", "shard": 0, "primary": false }

The answer names the shard's unassigned_info.reason (for example NODE_LEFT or INDEX_CREATED), a can_allocate verdict, and, per node, the deciders that said no. Read the explanation strings: they say things like "the node is above the high disk watermark" or "a copy of this shard is already allocated to this node". Called with no body, the API explains the first unassigned shard it finds.

The stand-in below is not Elasticsearch. It copies the idea of the deciders with two hand-written rules and prints a readable verdict, so you can practice reading this kind of output.

JS A tiny allocation explainer

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Only node-b can take the copy. If it were also full, the answer would be "no" and the fix would be to free disk or add capacity, not to retry.

Metrics worth watching

HTTP
GET /_nodes/stats/jvm,os,thread_pool,breaker,fs
GET /_cat/thread_pool/search,write?v&h=node_name,name,active,queue,rejected
GET /_nodes/hot_threads
  • JVM heap used (jvm.mem.heap_used_percent) sustained above about 85%, plus long GC pauses, signals memory pressure. Circuit breaker tripped counts tell you that Elasticsearch rejected requests to protect itself.
  • Thread pools: a growing rejected count on search or write means the node cannot keep up. Queues that are always full mean saturation, not bad luck.
  • File system: free space per data path (fs.total), because disk watermarks block allocation.
  • Hot threads shows what the busiest threads are doing right now (a heavy aggregation, a merge, a slow script). It is the tool for "CPU is at 100% and I do not know why".

Kibana Stack Monitoring or an external system (Prometheus with an exporter) should chart these over time; a single snapshot rarely shows a trend.

Slow logs

Slow logs record individual searches and indexing operations that exceed thresholds you choose. They are set per index and are off until you set them:

HTTP
PUT /products/_settings
{
  "index.search.slowlog.threshold.query.warn": "2s",
  "index.search.slowlog.threshold.query.info": "500ms",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.indexing.slowlog.threshold.index.warn": "5s"
}

Matching requests are written to the node's slow log files with the index, the time taken and, for searches, the query source. Use them to find the few queries that hurt, then fix them with the advice in query performance. Choose thresholds that catch real outliers, otherwise the log becomes noise.

Users, roles and role mappings in depth

The earlier section showed one role and one user. Real deployments add four things.

Role mappings and external realms

Users do not have to live inside Elasticsearch. A realm is an authentication source: the native realm (users stored in the cluster), file, LDAP, Active Directory, SAML, OpenID Connect and others. For external realms you write role mappings: rules that say "anyone whose group is X gets role Y".

HTTP
PUT /_security/role_mapping/search_team
{
  "enabled": true,
  "roles": ["product_search_reader"],
  "rules": {
    "all": [
      { "field": { "realm.name": "corp_saml" } },
      { "field": { "groups": "cn=search,ou=teams,dc=example,dc=com" } }
    ]
  }
}

Roles then follow group membership in your identity provider, with no passwords copied into Elasticsearch.

Document-level and field-level security

Field-level security (FLS) hides fields: a role with grant: ["name", "price"] never shows any other field in hits or aggregations. Document-level security (DLS) adds a query to every search the role runs, so only matching documents are visible. The query can use a templated user property such as {{_user.username}}, so one role can serve every tenant. These are subscription features, so check your license tier before relying on them.

The stand-in below models both rules on in-memory documents. It only illustrates what the role definition means; the real enforcement happens inside the cluster.

JS Simulating field and document security

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Generating certificates

For a self-managed cluster, the elasticsearch-certutil tool (in the bin folder) creates a certificate authority and node certificates:

Terminal
bin/elasticsearch-certutil ca --out elastic-stack-ca.p12
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 \
  --name node-1 --dns node-1.example.com --ip 10.0.0.11 \
  --out node-1.p12

Each node's elasticsearch.yml then points at its keystore, with the passwords kept in the Elasticsearch keystore rather than in the file:

YAML
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/node-1.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/node-1.p12

Include every DNS name and IP that clients use in the certificate, or hostname verification fails. Guard the CA key: anyone holding it can mint certificates your cluster trusts.

Audit logging

Audit logs record security events: failed logins, denied access, role changes, who ran which request. Turn it on with a setting; events are written as JSON lines to a file in the logs folder (availability depends on your license tier):

YAML
xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.include: [authentication_failed, access_denied, security_config_change]

Keep the include list short. Auditing every successful request on a busy cluster costs real I/O and produces more data than anyone reads.

Plugins and version upgrades in depth

Plugins

A plugin adds code to every node. Core plugins are maintained by Elastic and released with it; community plugins come from third parties, may lag behind releases, and run with full node privileges, so review them like any dependency. Analysis plugins are the commonest. The ICU plugin adds Unicode-aware tokenizing, normalization and collation, which helps with Asian languages, accents and sorting (see multilingual search):

Terminal
bin/elasticsearch-plugin install analysis-icu
bin/elasticsearch-plugin list

Rules to remember: a plugin build matches one exact Elasticsearch version, so it must be reinstalled on every node after each upgrade; every node needs the same plugins (a node missing a plugin that an index's analyzer uses cannot hold that index's shards); and installing needs a node restart, done one node at a time. Where you can, use a container image with the plugin baked in rather than hand-managed installs.

A safe upgrade path, step by step

  1. Run the upgrade assistant in Kibana (or call GET /_migration/deprecations). It lists deprecated settings, mappings and indexes created by too old a version.
  2. Fix deprecations and reindex any index flagged as too old, because a new major version cannot open indexes created two majors back: POST /_reindex into a new index, then switch the alias (see documents and indexes).
  3. Take a snapshot and test a restore.
  4. Upgrade to the latest minor release of your current major.
  5. Do the rolling upgrade one node at a time, in the order the upgrade guide gives for your versions (commonly non-master nodes first, master-eligible nodes last). For each node: restrict shard allocation (cluster.routing.allocation.enable: primaries), stop the node, upgrade it and its plugins, start it, re-enable allocation and wait for green.
  6. Upgrade clients and Kibana to match.

The stand-in below only prints such a plan for a small cluster. It cannot upgrade anything.

JS Printing a rolling upgrade plan

Press Run (or Ctrl + Enter) to see the console output here.

No output.

ES|QL: a first look

Besides the JSON query DSL, recent Elasticsearch versions have ES|QL, a piped query language. Each command transforms a table of rows and passes it on, much like a shell pipeline:

HTTP
POST /_query
{
  "query": """
    FROM logs-*
    | WHERE level == "error" AND @timestamp > NOW() - 1 hour
    | STATS errors = COUNT(*) BY service
    | SORT errors DESC
    | LIMIT 5
  """
}

ES|QL is convenient for analytics and ad hoc exploration, and its feature set grows with each release, so check the documentation of your version for what is supported. Remember that it has the same injection risk as any query language: never paste raw user text into a query string. ES|QL supports parameters (placeholders with separately passed values) for exactly this reason.

Query injection and safe user input

Search boxes feed text from strangers into your cluster. Problems come from two directions.

1. Building queries by string concatenation. If you assemble JSON by gluing text together, a user can close your string and add their own clauses:

JavaScript
// DANGEROUS (read-only): user text becomes part of the JSON structure
const body = '{ "query": { "match": { "name": "' + userText + '" } } }';

A user who types x" } }, "size": 10000, "query": { "match_all": {} } }// changes the meaning of the request. The fix is the usual one: build objects, not strings, and let the client serialize them:

JavaScript
const body = { query: { match: { name: userText } } };

Now the text is only ever a value. It cannot add keys.

2. Powerful query syntaxes. query_string accepts wildcards, regular expressions, field names (salary:>100000) and boolean operators. Exposing it to users lets them search fields you did not intend, run expensive queries (*a*b*c*) or crash the parser with a stray bracket. Prefer match or multi_match (you pick the fields), or simple_query_string if you want a safe subset of operators.

Other protections: cap the length of the input, cap size and page depth on your server rather than trusting client parameters, whitelist the fields users may sort or filter on, use a read-only role for the search service, and rate-limit the search endpoint. Do not return raw error messages from Elasticsearch to browsers; they can reveal index names and mapping details.

JS Sanitizing a search string and building a safe query

Press Run (or Ctrl + Enter) to see the console output here.

No output.

In the safe version, the attacker's quote characters stay inside one string value, size is clamped to 50, and the sort field must come from a whitelist (salary falls back to relevance). Escaping helps when query_string is unavoidable, but a well-behaved match query remains the better default. This is a stand-in written for teaching; the real escape rules for your version are listed in the query_string documentation, so rely on structured queries first.

JS A least-privilege permission check

Press Run (or Ctrl + Enter) to see the console output here.

No output.

This models the idea of the roles above: the web search key can read products and nothing else, even if someone steals it. The real security layer checks far more (clusters, fields, documents), but the principle is identical.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why are API keys often better than a shared username and password for an application?

    Each API key has its own restricted permissions and lifetime, so you can revoke one service's key without affecting others.
  2. 2. Your cluster has two replicas of every shard. Why do you still need snapshots?

    Replicas are live copies. A deleted index disappears from all of them. A snapshot is an independent, point-in-time backup.
  3. 3. Which approach is safest for passing a user's search text to Elasticsearch?

    Structured objects keep user text as data. It cannot add keys or clauses. query_string exposes a powerful syntax that most search boxes do not need.
  4. 4. What does a yellow cluster health status mean?

    Yellow means every primary is allocated (data is available), but at least one replica is not, so redundancy is reduced.

Summary

  • Layer your defenses: TLS, authentication, role-based authorization and a private network. Never expose an unsecured cluster.
  • Give applications narrow roles or API keys with expiry; keep the elastic superuser for administration.
  • Replicas are not backups. Use incremental snapshots to a repository, automate them, and test restores.
  • Watch cluster health, disk, heap and rejections; read the breaking changes and snapshot before upgrading.
  • ES|QL is a piped query language for analytics, with parameters for safe input.
  • Build queries as objects, whitelist fields and sorts, clamp sizes and lengths, and avoid exposing query_string.

Next: put it all together in Project: Product Search with Facets.