Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 43 ยท Cluster and Scale

Cross-Cluster Search and Replication

Connect Elasticsearch clusters: remote cluster setup, cross-cluster search with the cluster:index syntax, CCR leader and follower indexes, disaster recovery and snapshot restore.

One cluster does not always fit. You may run a cluster per region to keep data close to users, one per business unit, or a second cluster in another data center as a safety net. Elasticsearch has two features for connecting clusters: cross-cluster search (CCS) queries several clusters as if they were one, and cross-cluster replication (CCR) copies indexes from one cluster to another continuously.

The clusters cannot run on this site, so all requests below are read-only. The runnable examples are JavaScript stand-ins that model the behavior of CCS and CCR. They are simplified models, not Elasticsearch code.

Remote clusters

Both features start by teaching one cluster about another. You register the other cluster under a short alias, called a remote cluster. The modern way uses an API key and a dedicated port (9443 by default) rather than sharing the transport port 9300 with a full trust relationship:

HTTP
PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "eu_west": {
          "mode": "proxy",
          "proxy_address": "es-eu.example.com:9443",
          "skip_unavailable": true
        }
      }
    }
  }
}

GET /_remote/info

The nodes that talk to remotes need the remote_cluster_client role (see Shards, Replicas and Clusters). On the remote side, an administrator creates a cross-cluster API key that limits what the local cluster may search or replicate. The point of that key is least privilege: the local cluster can touch only the index names you allow.

skip_unavailable decides what happens when the remote is down: true returns partial results from the clusters that answered, while false fails the whole search.

To search a remote index, prefix the index with the cluster alias and a colon:

HTTP
GET /products,eu_west:products,us_east:products/_search
{
  "query": { "match": { "name": "trail shoe" } }
}

The request goes to the local cluster, which forwards a sub-request to each remote and merges the results. The response has a _clusters section that tells you how many clusters were successful, skipped or partial, and each hit's _index shows its origin (eu_west:products). Wildcards work too: *:products searches every configured remote.

Things to know:

  • Scores are not globally comparable. Each cluster computes BM25 statistics from its own data (see Relevance and BM25), so a document from a small cluster can score higher just because its term statistics differ. If ranking quality matters, use filters, sort by a field, or normalize in your application.
  • Latency is set by the slowest remote. Use timeout and skip_unavailable.
  • Minimize round trips. The default ccs_minimize_roundtrips: true sends one request per cluster and merges partial results, which is much faster over a WAN than contacting every shard individually.
  • Not everything is supported. Check the documentation for limits on features such as some aggregations and scroll across clusters.

A cross-cluster search stand-in

The stand-in below searches three small "clusters" with a simple term-frequency score, one of them unavailable. It shows prefixed index names, skip_unavailable, and why scores differ between clusters.

JS Searching clusters you do not own

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that the doc shoe shoe shoe sale from eu_west outranks the others purely on its own statistics. Real BM25 is more subtle, but the lesson holds: scores from different clusters are only loosely comparable.

Cross-cluster replication

CCS reads remote data in place. CCR instead copies it. An index on a leader cluster is replicated to a follower index on another cluster. Followers are read-only, and they pull changes by reading the leader's per-shard operation history, so replication keeps up with writes and lags only by a short time.

HTTP
PUT /products-copy/_ccr/follow
{
  "remote_cluster": "eu_west",
  "leader_index": "products"
}

GET /products-copy/_ccr/stats
POST /products-copy/_ccr/pause_follow
POST /products-copy/_ccr/unfollow

CCR needs a license level that includes it (check your subscription), soft deletes enabled on the leader index (the default for new indexes), and the remote cluster set up as above. You can also create an auto-follow pattern so that every new index matching logs-* on the leader is followed automatically.

Use cases

  • Disaster recovery. A follower cluster in another region has an almost current copy. If the primary region fails, you promote the follower: pause following, then unfollow to turn it into a normal writable index, and point your applications at it.
  • Read locality. Followers in each region serve local searches with low latency, while all writes go to one leader.
  • Centralized reporting. Replicate indexes from many small clusters into one analytics cluster.

CCR is not a backup. A bad delete on the leader is replicated too, which is why snapshots remain necessary (see Security, Backups and Operations).

Snapshots across clusters

A simpler option for many teams is a shared snapshot repository, for example one S3 bucket. The production cluster writes snapshots, and a second cluster registers the same repository (read-only) and restores from it. This is cheaper than CCR and fine when minutes or hours of delay are acceptable. Restores must go to a cluster whose version can read the snapshot.

A follower stand-in

This stand-in models a leader that applies operations with sequence numbers and a follower that pulls whatever it has not yet seen. After the leader "fails", the follower is promoted, and you can see exactly which operations were lost because of replication lag.

JS Leader, follower and lag

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The lost operations are the price of asynchronous replication. For most failover plans that is acceptable, and the lag can be monitored with the CCR stats API.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you search the index products on a remote cluster registered as eu_west?

    Cross-cluster search prefixes the index name with the remote cluster alias and a colon, for example eu_west:products.
  2. 2. What does skip_unavailable: true do?

    With true, an unreachable remote is skipped and the search succeeds with results from the other clusters. With false the whole search fails.
  3. 3. Which statement about CCR follower indexes is correct?

    Followers are read-only and replicate asynchronously. Promotion means pausing and unfollowing. Because a delete on the leader replicates too, CCR is not a backup.
  4. 4. Why can scores from different clusters be misleading to compare?

    BM25 uses term statistics such as document frequency, and each cluster calculates them from only its own documents.

Summary

  • Register other clusters as remote clusters with an alias, an address and a cross-cluster API key; nodes need the remote_cluster_client role.
  • Cross-cluster search uses cluster:index names, merges results, and can tolerate unreachable remotes with skip_unavailable.
  • Scores from different clusters are not directly comparable.
  • CCR replicates a leader index into a read-only follower, asynchronously, for disaster recovery, read locality and central reporting.
  • CCR is not a backup; keep snapshots, and consider a shared snapshot repository as a simpler cross-cluster option.

Next: Using Elasticsearch from Code, to talk to your clusters from an application.