Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 37 ยท Getting Started

Consul Architecture: Agents, Gossip and Raft

Understand how a Consul cluster works: server and client agents, the Serf gossip protocol, Raft consensus, quorum math, ports and the role of a datacenter.

A Consul cluster looks simple from outside: you ask a question and get an answer. Inside, three ideas make it work: agents on every machine, a gossip protocol that spreads membership news, and Raft consensus that keeps the servers' data identical. Knowing them explains Consul's behavior during failures, and it tells you how many servers to run.

As in the first lesson, the real Consul processes cannot run in the browser. The commands and configuration here are read-only; the two programs are Python stand-ins that model quorum math and gossip.

Agents: servers and clients

Every machine in the cluster runs the same program, consul agent, in one of two modes:

  • Server agents hold the catalog and key-value data, take part in Raft consensus and elect a leader. You run a small odd number of them (three or five in production).
  • Client agents hold no durable data. They run next to your applications, register local services, run health checks, and forward queries to a server.
flowchart TB
  subgraph DC["Datacenter dc1"]
    S1["Server (leader)"]
    S2["Server (follower)"]
    S3["Server (follower)"]
    C1["Client agent + web"]
    C2["Client agent + payments"]
  end
  C1 -->|RPC| S1
  C2 -->|RPC| S2
  S1 <-->|Raft| S2
  S1 <-->|Raft| S3
  C1 <-.->|gossip| C2

Clients and servers gossip with each other; only servers use Raft among themselves.

A client agent is deliberately cheap, so you can put one on every host. Applications talk to their local agent (127.0.0.1:8500 for HTTP, 127.0.0.1:8600 for DNS), which is why your code never needs to know where the servers are.

Gossip: who is in the cluster?

Agents learn about each other through Serf, a gossip protocol based on SWIM. Each agent periodically picks a few random peers and exchanges what it knows: joins, leaves and suspected failures. News spreads exponentially, so even a large cluster converges in a handful of rounds, and no central membership list can be a bottleneck.

There are two gossip pools:

  • LAN gossip joins every agent in one datacenter.
  • WAN gossip joins the servers of all datacenters, which enables multi-datacenter requests later in the course.

Gossip is encrypted when you set an encryption key. It is about membership and failure detection, not about storing data.

Raft: agreeing on data

Servers store the catalog and KV data using Raft, a consensus algorithm. One server is the leader. Writes go to the leader, which appends them to a log and replicates the entry to followers. A write is committed once a majority (a quorum) has stored it. If the leader fails, the followers elect a new one.

Quorum is the number that matters in operations:

Servers Quorum (majority) Failures tolerated
1 1 0
3 2 1
5 3 2
7 4 3

Two servers tolerate no failures (quorum is 2), so they are worse than one in terms of availability per machine. Even numbers add cost without adding tolerance. Beyond seven servers, replication slows writes with little benefit.

Python Stand-in: quorum and failure tolerance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Gossip speed in a model

The next stand-in is not Serf; it is a simplified push gossip in which each informed node tells two random peers per round. It shows why news reaches everyone in roughly log rounds. The random seed is fixed so the output is repeatable.

Python Stand-in: how fast news spreads

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that going from 10 to 5,000 nodes adds only a few rounds. That is why gossip suits large fleets.

The data path of a request

  1. Your app sends an HTTP or DNS query to its local client agent.
  2. The agent forwards it over RPC to a server.
  3. For default (consistent-leader) reads and all writes, the request is handled by the leader. Reads can be relaxed with query parameters to allow followers to answer (stale), trading freshness for speed.
  4. Results come back the same way.

Default ports

Port Purpose
8300 Server RPC (clients to servers, server to server)
8301 LAN gossip (TCP and UDP)
8302 WAN gossip (servers only)
8500 HTTP API and UI (8501 for HTTPS)
8502 gRPC API (8503 for gRPC with TLS)
8600 DNS (TCP and UDP)

Firewalls are the most common reason a new cluster refuses to form. If agents cannot join, check that 8301 is open in both TCP and UDP between all agents.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How many server failures can a 5-server Consul datacenter tolerate while still accepting writes?

    Quorum is 3 of 5, so up to 2 servers can fail.
  2. 2. What is the main job of a client agent?

    Clients keep no durable data; they sit next to applications and forward to servers.
  3. 3. Which protocol spreads membership and failure news among agents?

    Serf, a SWIM-based gossip protocol, handles membership and failure detection. Raft is for server data.
  4. 4. Why is running exactly 2 Consul servers a poor choice?

    With 2 servers the majority is 2, so one failure blocks writes; it tolerates no more failures than 1 server does.

Summary

  • Every machine runs an agent; servers store data and run Raft, clients are cheap forwarders.
  • Serf gossip handles membership and failure detection with LAN and WAN pools.
  • Raft commits a write when a majority of servers has it; use 3 or 5 servers.
  • Apps talk to their local agent on ports 8500 (HTTP) and 8600 (DNS).
  • Open 8301 TCP and UDP between agents or the cluster will not form.

Next: Installing Consul and Running a Dev Agent