Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 37 ยท Multi-datacenter

Consul Datacenters and WAN Federation

Connect Consul datacenters with WAN federation: LAN versus WAN gossip, the primary datacenter, joining with retry_join_wan, mesh gateways and ACL replication.

A single Consul cluster lives in one datacenter: a name (dc1) that groups servers and clients with fast, reliable networking between them, normally a cloud region or a physical site. Real systems often span several regions, for latency, disaster recovery or compliance. Consul supports this natively: you run one cluster per datacenter and connect them.

The product cannot run on this site. Configuration is read-only; the runnable examples are small models of the ideas.

Two gossip pools

Consul uses two gossip layers:

  • LAN pool (port 8301): every agent (servers and clients) in one datacenter. It detects failures quickly with tight timings, and is what lets clients find servers without configuration.
  • WAN pool (port 8302): only the servers, of all datacenters, with timings tuned for the slower internet.

Each datacenter elects its own Raft leader. Writes go to the local leader, so a datacenter keeps working if its neighbors vanish. What is shared between datacenters is discovery and routing: one datacenter can ask another where a service lives. The catalog and key-value data themselves are not replicated between datacenters (ACLs and config entries are, from the primary).

flowchart LR
  subgraph dc1
    S1["servers (Raft)"]
    C1["clients"]
  end
  subgraph dc2
    S2["servers (Raft)"]
    C2["clients"]
  end
  C1 --- S1
  C2 --- S2
  S1 <-- "WAN gossip 8302, RPC 8300" --> S2

Clients stay inside their LAN; only servers cross the WAN.

Joining two datacenters

Give each datacenter a unique name and the same primary:

HCL
# dc1 servers
datacenter         = "dc1"
primary_datacenter = "dc1"
server             = true
retry_join_wan     = ["10.1.0.10", "10.1.0.11"]   # or addresses in dc2
HCL
# dc2 servers
datacenter         = "dc2"
primary_datacenter = "dc1"
server             = true
retry_join_wan     = ["10.1.0.10"]                # reach a dc1 server

You can also join by hand once: consul join -wan <address of a server in the other datacenter>. Then confirm:

Terminal
consul members -wan
consul catalog datacenters

consul catalog datacenters lists the federated datacenters ordered by network distance, measured with round-trip estimates from Serf's network coordinates.

Requirements:

  • Unique datacenter names.
  • Server-to-server reachability on 8300 (TCP) and 8302 (TCP and UDP) across the WAN, directly or through mesh gateways.
  • Consistent encryption: ACLs and TLS should be configured similarly, and WAN gossip can use its own keyring (consul keyring -list shows both pools).

The primary datacenter

One datacenter is the primary. Secondary datacenters rely on it for:

  • ACL replication: tokens, policies and roles created in the primary appear in the secondaries (set acl.enable_token_replication = true on secondaries and give them a replication token).
  • Config entries: service-defaults, proxy-defaults, intentions and the like replicate from primary to secondaries.
  • The mesh CA root: the primary's CA signs, and secondaries obtain intermediates.

If the primary is unreachable, existing tokens keep working in secondaries for a while (cached), but you cannot create global tokens or change replicated config. Plan for the primary's availability and have a runbook for promoting a secondary (a deliberate, careful operation).

Mesh gateways for federation

Direct server-to-server reachability across datacenters often is not available: overlapping private IP ranges, no VPN, strict firewalls. Mesh gateways are proxies at the edge of each datacenter that forward encrypted traffic without decrypting it. With WAN federation over mesh gateways you enable:

HCL
connect {
  enable_mesh_gateway_wan_federation = true
}

The servers then send their cross-datacenter RPC and gossip through the gateways instead of directly. Service mesh traffic between datacenters also flows gateway to gateway:

HCL
Kind = "proxy-defaults"
Name = "global"

MeshGateway {
  Mode = "local"
}

local sends cross-datacenter traffic through the gateway in the caller's datacenter first, which keeps firewall rules simple; remote sends directly to the remote gateway; none disallows gateway use.

Python Stand-in: federation checklist

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Failure behavior

  • Network partition between datacenters: each side keeps serving local traffic; cross-datacenter lookups fail or fall back (see failover).
  • Lost primary: replicated data stays readable in secondaries; writes to global ACLs and config entries stop until it returns.
  • One datacenter has no leader: only that datacenter is impaired; others carry on.

Because there is no cross-datacenter consensus, Consul's multi-datacenter model favors availability and local latency over global consistency. Do not use it as a globally consistent database.

Latency and coordinates

Servers continuously measure round-trip times using network coordinates. consul rtt dc1-server-1 dc2-server-1 estimates the latency between two nodes, and prepared queries can use the same data to find the "nearest" service instance. Good numbers matter: WAN gossip timings assume tens to low hundreds of milliseconds.

Python Stand-in: nearest datacenter by round-trip time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Two datacenters with the same name. They merge into a confusing single pool; names must be unique.
  • Opening 8301 across the WAN. Only server WAN ports (8300, 8302) need to cross; client LAN gossip must stay inside one datacenter.
  • Missing replication token. Secondaries cannot replicate ACLs and report "ACL not found" errors.
  • Treating datacenter as availability zone. All servers of one datacenter should be reachable with low latency; a stretched cluster across distant regions causes Raft instability. Use several datacenters or availability zones within a region instead.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which agents join the WAN gossip pool?

    WAN gossip connects the servers of different datacenters; clients stay in their LAN pool.
  2. 2. What is replicated from the primary datacenter to secondaries?

    Catalog and KV data stay local; global ACL objects and config entries replicate.
  3. 3. What does retry_join_wan do?

    It lists addresses of servers in remote datacenters to connect to the WAN pool.
  4. 4. Why use mesh gateways for federation?

    Gateways carry the cross-datacenter traffic through a small number of reachable endpoints.
  5. 5. What happens to dc2's local services if dc1 becomes unreachable?

    Each datacenter has its own Raft cluster, so local operation continues.

Summary

  • Each datacenter is its own Raft cluster; servers of all datacenters share a WAN gossip pool.
  • Join with retry_join_wan or consul join -wan; verify with consul members -wan and consul catalog datacenters.
  • The primary datacenter provides ACL, config entry and CA replication.
  • Mesh gateways let federation work without direct server connectivity; local operation survives WAN failures.

Next: Cross-Datacenter Discovery and Failover