Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Operations

Consul Raft, Quorum and Autopilot

Understand how Consul servers elect a leader with Raft, why you need 3 or 5 servers, how quorum and fault tolerance work, and what Autopilot automates.

Everything in Consul's catalog, key-value store, ACLs and config entries lives on the servers, and the servers must agree on every change. They do it with the Raft consensus protocol. Understanding Raft, quorum and Autopilot is the difference between a cluster that survives a bad night and one that stops accepting writes.

The product cannot run on this site. Commands are read-only, and the runnable examples are stand-in calculators and simulators.

Leader, followers and the log

At any moment one server is the leader. All writes go to the leader, which appends them to a replicated log and sends them to the followers. A write is committed once a majority of servers have stored it, and only then does the leader tell the client it succeeded. Followers apply committed entries to their own state, so all servers converge on the same data.

Reads can be served in different consistency modes:

  • default: the leader answers after checking that it still leads. Almost always up to date.
  • consistent: the leader verifies leadership with a quorum first. Strongest, slowest.
  • stale (?stale): any server answers, even a follower. Fastest, may lag slightly, and works even when there is no leader. Use it for read-heavy lookups that tolerate small delays, such as DNS (dns_config { allow_stale = true }).

Client agents are not part of Raft. They forward requests to servers.

Elections

Followers expect regular heartbeats from the leader. If one stops hearing them, it becomes a candidate and asks the others for votes. A candidate that wins a majority becomes leader. While there is no leader, writes and default reads fail with "No cluster leader" until a new one is elected, usually in seconds.

Quorum

Quorum is a majority of the servers: floor(N/2) + 1. With quorum you can elect a leader and commit writes; without it the cluster is read-only at best (stale reads) and stuck.

Servers (N) Quorum Failures tolerated
1 1 0
2 2 0
3 2 1
4 3 1
5 3 2
6 4 2
7 4 3

Two lessons come out of this table:

  1. Even numbers add cost, not safety. Four servers tolerate one failure, the same as three.
  2. Use 3 or 5. Three servers tolerate one failure, five tolerate two. More than five slows every write, since a bigger majority must acknowledge, and gives little benefit. Spread servers across failure domains (availability zones) so a single failure takes out at most one.
Python Stand-in: quorum and fault tolerance calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Losing quorum

If quorum is lost, for example two of three servers crash, the remaining server cannot elect a leader and the cluster halts writes. The fix is to bring failed servers back. If they are gone for good, an operator can recover with a peers.json file in the data directory (it lists the surviving servers) and a restart, which is a last resort documented in the Raft recovery guide. Never run two servers with the same node ID, and never restore a partial set of servers from different snapshots.

Inspect the cluster:

Terminal
consul operator raft list-peers

The output shows each server's node, ID, address, state (leader, follower), whether it votes and its Raft protocol version. Also useful: consul members (gossip view), consul info (look at the raft: and serf_lan: sections, such as commit_index and last_log_index).

Python Stand-in: leader election simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The election winner differs with different random seeds, so read the pattern rather than the specific server numbers: leaders survive up to the tolerance and elections stop when alive servers drop below quorum.

Autopilot

Autopilot keeps the server set healthy with minimal manual work. In open-source Consul it provides:

  • Dead server cleanup: removes failed servers from the Raft peer set after they have been unhealthy for a while, so they do not count against quorum math and you can add replacements.
  • Server health: classifies each server as healthy or not using LastContactThreshold (200ms by default), MaxTrailingLogs (250) and ServerStabilizationTime (10 seconds). A new server is promoted to a voter only after it has been healthy and caught up for the stabilization time.
  • Operator view: consul operator autopilot state shows the details.

Inspect and tune:

Terminal
consul operator autopilot get-config
consul operator autopilot set-config -cleanup-dead-servers=true -last-contact-threshold=200ms
consul operator autopilot state

Enterprise adds redundancy zones, automated upgrades and read replicas (non-voting servers), which are beyond the open-source feature set.

Raft performance and sizing

  • Disk latency matters most. Every commit is an fsync on a majority of servers. Use fast SSDs and keep the data directory on its own volume.
  • Network latency between servers should be low and stable (single-digit milliseconds in one region). Stretching one cluster across distant regions causes slow commits and spurious elections.
  • Memory holds the whole state in RAM; size for the catalog and KV data plus headroom for snapshots.
  • Watch consul.raft.commitTime, consul.raft.leader.lastContact and consul.autopilot.healthy (see Telemetry).
Python Stand-in: sizing check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Running two servers "for redundancy": you now need both to be up, which is worse than one.
  • Putting all servers in one failure domain, then losing the rack.
  • Using more than five servers hoping for speed. Add client agents for scale instead; servers do not need to grow with the number of services.
  • Treating force-leave as removal from Raft. consul force-leave changes gossip state; use consul operator raft remove-peer for a failed server that Autopilot has not cleaned up.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the quorum for a 5-server cluster?

    Quorum is floor(5/2) + 1 = 3, so two servers may fail.
  2. 2. Why is a 4-server cluster not better than a 3-server cluster for fault tolerance?

    Quorum for 4 is 3, so one failure is tolerated, the same as for 3 servers.
  3. 3. Which read mode can be answered by a follower and works without a leader?

    Stale reads trade freshness for availability and speed.
  4. 4. What does Autopilot's dead server cleanup do?

    It keeps the voter list accurate so replacement servers can be added.
  5. 5. Which command shows the Raft peers and who leads?

    list-peers prints each server's state, such as leader or follower, and whether it votes.

Summary

  • Servers replicate a log with Raft: one leader takes writes, a majority commit them.
  • Quorum is floor(N/2)+1; run 3 or 5 servers across failure domains, never an even count or two.
  • Stale reads trade freshness for availability; losing quorum halts writes until servers return or you recover manually.
  • Autopilot cleans up dead servers, tracks health and promotes new servers safely.

Next: Telemetry, Logs and Monitoring