Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 43 ยท Cluster and Scale

Shards, Replicas and Clusters

Understand Elasticsearch clusters: primary and replica shards, document routing by hash, node roles, master election, shard allocation, and green, yellow or red health.

One computer cannot hold or search unlimited data. Elasticsearch solves this by splitting each index into pieces called shards and spreading the pieces over several machines. Several machines working together form a cluster. This design gives you two things at once: more data and speed (many shards search in parallel) and safety (copies of every shard live on different machines).

This lesson explains the moving parts: primary and replica shards, how a document picks its shard, what the different node roles do, how the cluster agrees on who is in charge, and what green, yellow and red health really mean. The real cluster cannot run on this site, so the requests are read-only. Two JavaScript stand-ins model the routing and the allocation logic. They are simplified models, not Elasticsearch code.

The building blocks

Node One running Elasticsearch process.
Cluster Nodes that share one state.
Shard A slice of an index, a full Lucene index.
Replica A copy of a primary shard.
1 / 2

Primary and replica shards

When you create an index you decide how many primary shards it has. Each primary holds a distinct part of the documents. A replica shard is an exact copy of one primary, kept on a different node.

JSON
PUT /orders
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  }
}

This index has 3 primaries and 3 replicas, so 6 shard copies in total. Replicas do two jobs. They protect your data: if the node holding a primary dies, its replica on another node is promoted. And they add read capacity, because searches can be served by either the primary or a replica.

Two rules to remember. First, number_of_shards is fixed when the index is created. To change it you use the split, shrink or reindex APIs, never an in-place update. Second, number_of_replicas can be changed at any time. A replica is never placed on the same node as its primary, because a copy next to the original would protect nothing.

Each shard is a complete Lucene index of its own, with its own segments (Segments, Refresh and Near-Real-Time Search). That means every shard has overhead in memory and file handles. Many tiny shards waste resources, and a handful of huge shards recover slowly. A common rule of thumb is shards of tens of gigabytes, but measure with your own data.

How a document finds its shard

When you index a document, Elasticsearch needs to decide, quickly and without asking anyone, which primary receives it. The formula is:

shard = hash(routing) % number_of_primary_shards

By default routing is the document _id. The real engine uses a Murmur3 hash, which spreads IDs evenly. Because the same ID always hashes to the same shard, a GET /orders/_doc/42 can go straight to one shard, without searching all of them. A search query, on the other hand, must ask every shard (the coordinating node sends the query to one copy of each shard and merges the answers).

The formula also explains why the number of primaries is fixed: change the divisor and most documents would belong to a different shard than the one that stores them.

You can pass your own routing value (?routing=customer-17) so that all documents of a customer land on the same shard, and a search with the same routing then touches only that shard. This is powerful for multi-tenant data, but a very large customer can create a "hot" shard, so use it deliberately.

JS Route documents to shards by hash

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The twelve documents are spread across the three shards (the exact split depends on the hash), and a large share of them would map to a different shard if the divisor changed, which is why Elasticsearch never allows it in place. With a routing value, every document of the customer reports the same shard.

Nodes and roles

A node is one running Elasticsearch process, and it can have several roles.

  • Master-eligible nodes can take part in electing the master, the node that manages the cluster state: which indexes exist, their mappings and settings, and where every shard lives. The master does not handle your documents. If it is overloaded, the cluster feels sluggish, so keep it light.
  • Data nodes store shards and do the indexing and searching work. Larger clusters use tiers (data_hot, data_warm, data_cold) that match the lifecycle in Geo Queries and Time-Series Data.
  • Ingest nodes run ingest pipelines (parsing, enriching) before documents are indexed.
  • Coordinating work happens on every node: the node that receives your request forwards it to the right shards and merges the answers. A node with no other roles acts as a dedicated coordinator.

For production, a typical small cluster has three master-eligible nodes that also hold data, or three small dedicated masters plus data nodes.

Master election in one paragraph

The cluster must have exactly one master at a time, even if the network splits. Elasticsearch requires a quorum: a majority of the voting master-eligible nodes must agree on the master. With three voters, any two form a majority, so one node can fail or be cut off and the cluster still elects a master. With only two voters, losing one leaves no majority, and with four voters you survive no more failures than with three. That is why the advice is an odd number, normally three. The side of a network split that has no majority stops accepting changes instead of creating a second, conflicting master.

Shard allocation and health

The master decides where each shard copy runs, which is called allocation. Its rules include: a replica never shares a node with its primary, load should be roughly balanced, nodes must have free disk space (watermark settings stop allocation when disks fill up), and you can add your own constraints such as zones or hot/warm tiers. When a node leaves, the master promotes replicas to primaries and creates new replicas elsewhere; when a node joins, shards rebalance onto it.

You see the outcome in the cluster health:

JSON
GET _cluster/health?pretty
{
  "status": "yellow",
  "number_of_nodes": 1,
  "active_primary_shards": 3,
  "unassigned_shards": 3
}
  • green: every primary and every replica is assigned.
  • yellow: all primaries are assigned, but some replicas are not. Searches and writes work; you just have less redundancy. A single-node cluster with number_of_replicas: 1 is permanently yellow, because the replica has nowhere to go.
  • red: at least one primary is unassigned. Part of your data is unavailable, and searches may return partial results.

GET _cat/shards?v lists every shard and its node, and GET _cluster/allocation/explain tells you why a particular shard is unassigned. Start there when health is not green.

JS Allocation, node failure and health colors

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Follow the scenarios in the output. With three nodes the replicas sit away from their primaries and health is green. When one node fails, replicas are promoted and new copies are built on the two survivors, so health is green again. After a second failure only one node remains, so the replicas have nowhere to go and the status is yellow. A single node with a replica configured is yellow from the start. With no replicas at all, losing a node loses the primaries it held, and the status turns red until the node comes back.

Configuring roles, awareness and watermarks

Roles are set per node in elasticsearch.yml with node.roles. A node with no setting gets every role; as soon as you list roles, it gets only those.

YAML
# a dedicated master-eligible node
node.roles: [ master ]

# a hot-tier data node that also runs ingest pipelines
node.roles: [ data_hot, data_content, ingest ]

# a pure coordinating node: an empty list
node.roles: [ ]

Besides master, data_*, ingest and the coordinating (empty) case, there are ml (machine learning jobs), remote_cluster_client (needed to talk to other clusters, see Cross-Cluster Search and Replication) and transform. data_content holds ordinary indexes such as a product catalog; the hot, warm and cold tiers hold time-series data as it ages.

Allocation awareness

If two shard copies sit in the same rack or availability zone, one power cut kills both. Shard allocation awareness tells the master about your topology so it spreads copies across it:

YAML
# on each node: which zone am I in?
node.attr.zone: zone-a

# cluster-wide: spread copies across the zone attribute
cluster.routing.allocation.awareness.attributes: zone

With awareness on, a primary in zone-a gets its replica in another zone whenever one has a node. Forced awareness (cluster.routing.allocation.awareness.force.zone.values: zone-a,zone-b) goes further: if a whole zone vanishes, the master will not pile all missing replicas into the surviving zone, which could overload it.

Allocation filtering

Filtering pins or bans indexes on nodes by attribute, name or IP. A classic use is draining a node before maintenance:

HTTP
PUT /_cluster/settings
{ "persistent": { "cluster.routing.allocation.exclude._name": "node-3" } }

The master moves every shard off node-3, after which you can stop it with no yellow period. Per-index settings such as index.routing.allocation.require.data: warm do the same for one index.

Disk watermarks

Allocation also respects disk space. Three thresholds, by default:

Setting Default What happens
low watermark 85% used no new shards are allocated to this node
high watermark 90% used the master tries to move shards away
flood_stage 95% used indexes with a shard here become read-only for writes

Free space or add a node, then clear the block (recent versions release it automatically once space is available). Use GET /_cluster/allocation/explain to ask why a particular shard is unassigned.

An allocation simulator

This stand-in follows the rules above: copies of the same shard never share a node, copies prefer different zones, and nodes above the low watermark are skipped. It is a simplified model, not the real allocator.

JS Zones, watermarks and a drained node

Press Run (or Ctrl + Enter) to see the console output here.

No output.

n2 is above the low watermark, so it receives nothing. Each shard's two copies land in different zones, and when n3 is drained its replica moves to n4, which is still in a different zone from the primary on n1.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which setting can you NOT change on an existing index with a simple settings update?

    The primary shard count is baked into document routing (hash % shards). Replicas and refresh interval are dynamic settings.
  2. 2. A document is indexed without a routing value. What decides its shard?

    _id is the default routing value. The same ID always maps to the same primary shard.
  3. 3. Why is a single-node cluster with one replica configured yellow?

    Yellow means all primaries are assigned but some replicas are not. A replica next to its own primary would protect nothing, so it stays unassigned.
  4. 4. Why do production clusters usually have three master-eligible nodes?

    Quorum means a majority of voters. Three voters tolerate one failure; two voters tolerate none, and four voters tolerate no more failures than three.

Summary

  • An index is split into primary shards; each primary can have replicas on other nodes.
  • A document's shard is hash(routing) % number_of_primary_shards, which is why the primary count is fixed after creation.
  • Replicas give redundancy and read capacity and are never on the same node as their primary.
  • Nodes have roles; the elected master manages cluster state, and an odd number of master-eligible nodes (usually three) keeps elections safe.
  • Green means all shards assigned, yellow means missing replicas, red means missing primaries; use _cat/shards and the allocation explain API to investigate.

Next: Query Performance and Tuning.