Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Multi-datacenter

Cross-Datacenter Discovery and Failover in Consul

Find services in other Consul datacenters with DNS and the HTTP API, use prepared queries for nearest and failover lookups, and fail over mesh traffic with resolvers.

Once datacenters are federated, a service in dc1 can find and call services that run in dc2. This lesson shows the ways to ask: DNS names with a datacenter label, the dc parameter on the API, prepared queries that fail over automatically, and mesh failover through resolvers.

The product cannot run on this site. Commands and entries are read-only; the runnable examples are stand-ins for the decision logic.

Asking another datacenter directly

By default every lookup targets the local datacenter. Name another one explicitly.

DNS: insert the datacenter between service and the domain:

Terminal
dig @127.0.0.1 -p 8600 web.service.consul        # local
dig @127.0.0.1 -p 8600 web.service.dc2.consul    # dc2

HTTP API: add ?dc=dc2:

Terminal
curl "http://127.0.0.1:8500/v1/health/service/web?passing&dc=dc2"

CLI: consul catalog services -datacenter dc2, consul kv get -datacenter dc2 config/x.

The request goes to a local server, which forwards it over the WAN to a server in the target datacenter. With ACLs, the token must be valid in the target datacenter, so use a global token (not -local).

Why a hard-coded datacenter is brittle

If your application always asks web.service.dc2.consul, it breaks when dc2 is down, and ignores a healthy local copy. You usually want "local if healthy, otherwise the nearest healthy copy elsewhere". Two features give you that.

Prepared queries

A prepared query is a saved, named lookup with built-in logic. Create one with the HTTP API (there is no dedicated CLI for creating them):

Terminal
curl --request POST --data @query.json \
  http://127.0.0.1:8500/v1/query
JSON
{
  "Name": "web-failover",
  "Service": {
    "Service": "web",
    "OnlyPassing": true,
    "Failover": {
      "NearestN": 2,
      "Datacenters": ["dc3"]
    },
    "Near": "_agent"
  }
}

What this does:

  1. Look for passing web instances in the local datacenter.
  2. If there are none, try the 2 nearest datacenters (ordered by round-trip time), then dc3.
  3. Near: "_agent" sorts instances by distance from the agent asking.

Clients consume it through DNS: web-failover.query.consul. Since the answer is computed on each lookup, applications need no failover code at all. Prepared queries need query_prefix ACL rules to create, and the DNS interface reads them with the anonymous token unless configured otherwise, so grant that carefully.

Python Stand-in: prepared-query failover

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Failover for service mesh traffic

In the mesh, failover belongs in a service-resolver (see Traffic Routing):

HCL
Kind = "service-resolver"
Name = "api"

Failover = {
  "*" = {
    Datacenters = ["dc2", "dc3"]
  }
}

If no healthy local api instance exists, the sidecar sends requests to api in dc2, then dc3. The calls pass through mesh gateways, so set a gateway mode:

HCL
Kind = "proxy-defaults"
Name = "global"

MeshGateway {
  Mode = "local"
}

Another option is Redirect, which permanently points a service at another datacenter (for example during a migration), and upstreams can also name a datacenter directly: datacenter = "dc2" on the upstream, so localhost:9191 always reaches dc2's api.

Requirements for cross-datacenter mesh traffic:

  • A mesh gateway in each involved datacenter, registered with consul connect envoy -gateway=mesh -register.
  • Intentions allowing the source (intentions replicate from the primary).
  • Matching service names; the destination datacenter needs a healthy sidecar-enabled service.
  • Enough capacity: failed-over traffic doubles load elsewhere.
Python Stand-in: resolver failover order for mesh traffic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Designing for failover

  • Plan capacity: if each of two datacenters runs at 70% load, failing one over overloads the other. Run at 50% each, or have a third.
  • Check dependencies: a failed-over service still needs its database. A stateless service in dc2 talking to a database in dc1 may add latency or break entirely.
  • Test regularly: scale a service to zero in one datacenter on purpose (a game day) and watch where traffic goes.
  • Avoid flapping: failing back immediately when the primary recovers can overload it. Resolvers fail back automatically when instances become healthy, so ensure warm-up (readiness checks) works.
  • Watch the WAN: failover adds round-trip latency to every call. Cache or reduce chattiness.

Multi-datacenter key-value and sessions

Each datacenter has its own KV store, so consul kv put -datacenter dc2 writes only there. Replicate selected keys with the separate consul-replicate pattern or a configuration management tool; do not expect automatic replication. Locks and sessions are also per datacenter, so a lock in dc1 does not exclude a client in dc2.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which DNS name asks the dc2 datacenter for web?

    The datacenter label goes between service and the domain: web.service.dc2.consul.
  2. 2. What does a prepared query's Failover with NearestN do?

    Nearest is determined by network round-trip estimates between servers.
  3. 3. Where do you configure mesh failover to another datacenter?

    The resolver config entry lists datacenters to try when local instances are unhealthy.
  4. 4. What carries mesh traffic between datacenters when direct connectivity is limited?

    Mesh gateways forward encrypted traffic between datacenters.
  5. 5. Is the KV store replicated automatically between datacenters?

    KV data is local; ACLs and config entries are the replicated objects.

Summary

  • Query another datacenter by adding its name to DNS, ?dc= on the API or the CLI's -datacenter flag.
  • Prepared queries compute failover for DNS clients; resolvers do it for mesh traffic through mesh gateways.
  • Plan capacity, dependencies and game-day tests; the KV store and sessions stay per datacenter.

Next: Cluster Peering