Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 38 ยท Swarm Architecture

Raft Consensus and Quorum

Learn how Docker Swarm managers use the Raft algorithm to agree on cluster state, what quorum means, and how to recover when too many managers fail.

Several managers must agree on one version of the truth: which services exist, which tasks run where. Disagreement would be disastrous (two managers scheduling the same task twice, for instance). Swarm solves this with the Raft consensus algorithm. You do not need to know Raft's internals to run Swarm, but you must understand quorum, because it decides how many managers you need and what happens when some fail.

The basic idea of Raft

  • Managers keep an ordered log of every change to the cluster (create service, update node, add secret).
  • One manager is the leader. All changes go through it.
  • The leader appends a change to its log and sends it to the followers. The change is committed when a majority of managers have stored it.
  • If the leader stops sending heartbeats, followers start an election. A candidate that gets votes from a majority becomes the new leader.
sequenceDiagram
  participant C as You
  participant L as Leader
  participant F1 as Manager 2
  participant F2 as Manager 3
  C->>L: docker service create
  L->>F1: append entry
  L->>F2: append entry
  F1-->>L: stored
  L->>L: majority reached, commit
  L-->>C: service created

A change is committed once a majority of managers has it, not all of them.

Quorum

Quorum is the majority: floor(N / 2) + 1 out of N managers. The cluster can change state only while a quorum of managers can talk to each other.

Managers (N) Quorum Tolerated failures
1 1 0
2 2 0
3 2 1
4 3 1
5 3 2
6 4 2
7 4 3

Look at two rows: two managers tolerate zero failures, which is no better than one (and worse, since either dying stops writes). Four tolerate the same as three. That is why you use odd numbers.

What happens without quorum

If you lose quorum (say two of three managers die), the cluster becomes read-only in a harsh way:

  • Running containers keep running. Workers do not need the managers to keep their existing tasks alive.
  • You cannot change anything. docker service create, update, scale, node changes fail with an error saying the swarm has no leader or lost quorum.
  • Failed tasks are not replaced. No scheduler is running, so there is no self-healing.

This is a surprise to many people: a quorum loss is not an immediate outage, but it removes all control until it is fixed.

Network partitions

Suppose five managers are split by a network failure into groups of 3 and 2. The group of 3 has quorum and carries on. The group of 2 cannot commit anything and stays idle. When the network heals, the minority catches up from the leader. There is never a "split brain" with two active leaders, because both cannot have a majority.

Recovering from lost quorum

Best case: bring the failed managers back. Their logs are on disk, and they rejoin and catch up.

If they are lost for good, force a new cluster from a surviving manager that has the data:

CONSOLE
$ docker swarm init --force-new-cluster --advertise-addr <this-node-ip>

This node becomes the only manager with the existing services, secrets and configs intact. Then re-add managers with docker swarm join-token manager. Before doing it, back up /var/lib/docker/swarm (stop Docker on that node first for a consistent copy), as covered in Operations.

If a manager is gone permanently and quorum still holds, simply docker node demote (if reachable) and docker node rm --force <id>.

Keep managers healthy

  • Low latency between managers: Raft heartbeats are in the order of a second, so avoid managers far apart over slow links.
  • Fast disks: the log is written synchronously. Slow storage makes elections flap.
  • Dedicated managers in bigger clusters, so a busy container cannot starve them.
  • Never run an even number of managers for long.
  • Don't reboot half your managers at once. Update them one at a time and wait for Reachable between each.

Stand-in: quorum calculator

Use this to explore the table above. It is plain arithmetic, no Docker involved.

Python Stand-in: quorum calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: can the cluster still decide?

This stand-in checks whether a given set of reachable managers still holds quorum, including a partition.

Python Stand-in: partition check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The 2/2 split with four managers shows the weakness of even numbers: neither side can proceed.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. How many managers form a quorum in a swarm with 5 managers?

    Quorum is floor(5/2) + 1 = 3.
  2. 2. Two of three managers fail. What happens to running containers?

    Workers keep their tasks alive, but without quorum nothing can be changed and failed tasks are not replaced.
  3. 3. Why is an even number of managers discouraged?

    Four managers tolerate one failure, the same as three, and a 2/2 partition leaves no side with a majority.

Summary

  • Managers replicate a log with Raft; a change commits when a majority has it.
  • Quorum is floor(N/2) + 1; use 3, 5 or 7 managers.
  • Without quorum the cluster cannot change but running tasks continue.
  • Recover with the old managers, or docker swarm init --force-new-cluster as a last resort.

Next: Node Management