Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 38 ยท Operations and Alternatives

Backup and Disaster Recovery for Docker Swarm

Back up Swarm state from a manager, restore it, recover from lost quorum with --force-new-cluster, and protect volumes and data with a tested disaster plan.

Swarm's cluster state (services, networks, secrets, configs, node membership) lives in the Raft log on the managers. Your application data lives in volumes and databases, which Swarm does not back up for you. A recovery plan needs both, and it needs to be rehearsed before the bad day.

Swarm is not available on this site, so every command below is read-only text. The Python blocks are stand-ins that model quorum arithmetic and a backup checklist.

What to back up

  1. Swarm state: the directory /var/lib/docker/swarm on a manager (Raft log, certificates, keys).
  2. Definitions: stack files, config files and scripts, in Git. If your stacks are all in Git, even a total loss can be rebuilt by docker swarm init plus docker stack deploy.
  3. Secrets' source values: secrets cannot be read back, so keep originals in a password manager or vault.
  4. Autolock key, if enabled (Protecting Swarm Secrets): stored apart from the backup.
  5. Data: volumes, databases, uploaded files. Use the database's own backup tool (pg_dump, mysqldump, snapshots) and copy the result off the cluster.

Backing up the swarm state

Back up from a manager. To get a consistent copy, stop Docker on that manager briefly, which is safe only if you have at least three managers (the others keep quorum):

Terminal
sudo systemctl stop docker
sudo tar czvf swarm-backup-$(date +%F).tar.gz -C /var/lib/docker swarm
sudo systemctl start docker

Do this on one manager at a time, never on all, and store the archive off-site and encrypted. In a swarm with one manager you cannot do this without downtime of the control plane; running tasks keep running, but nothing can be changed until Docker is back.

Restoring

Restoring to a new or rebuilt manager:

Terminal
sudo systemctl stop docker
sudo rm -rf /var/lib/docker/swarm
sudo tar xzvf swarm-backup.tar.gz -C /var/lib/docker
sudo systemctl start docker
docker swarm init --force-new-cluster --advertise-addr 10.0.0.5

--force-new-cluster tells this node to start a cluster from its own copy of the Raft data, discarding the old member list. The node becomes the only manager; then you re-join or promote other nodes (docker node promote). Workers that still hold valid certificates reconnect; stale nodes may need docker swarm leave --force and a fresh join. Existing services restart from the restored definitions. Data in volumes is not part of this backup.

Losing quorum

Raft needs a majority of managers online to make changes. With N managers the majority is floor(N/2)+1:

Managers Majority Failures tolerated
1 1 0
3 2 1
5 3 2
7 4 3

Use 3 or 5 managers, spread across failure domains (racks or zones). Even numbers add cost without extra tolerance (4 managers tolerate 1 failure, the same as 3). If you lose quorum permanently (for example two of three managers are destroyed), the surviving manager rejects changes. Recovery:

  1. On the surviving manager, run docker swarm init --force-new-cluster --advertise-addr <ip>.
  2. Join replacement managers: docker swarm join-token manager and run the printed command on new hosts.
  3. Verify with docker node ls and docker service ls.

Running tasks on workers keep serving traffic during a quorum loss; you simply cannot schedule or update anything.

flowchart TD
  L["Quorum lost"] --> Q{"Any manager<br/>data intact?"}
  Q -->|yes| F["docker swarm init<br/>--force-new-cluster"]
  Q -->|no| B["Restore swarm backup<br/>or redeploy from Git"]
  F --> J["Join new managers"]
  B --> J

Recovery always ends with re-adding managers and verifying services.

Planning for disaster

  • Define RPO and RTO. How much data may you lose, and how fast must you be back? They decide backup frequency and whether you need a standby cluster.
  • Automate backups and verify them by restoring into a scratch cluster on a schedule. An untested backup is a hope.
  • Replicate data off-site. Cluster state is small; database dumps and object stores matter more.
  • Keep infrastructure as code (Terraform or scripts) so nodes can be recreated.
  • Write the runbook and keep a copy outside the cluster, since you will not be able to read the wiki hosted on the dead cluster.
  • Practice node loss: drain a node (docker node update --availability drain node-2), remove it, and confirm tasks reschedule.

Stand-in: quorum arithmetic

Python Stand-in: how many managers can fail?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: is this backup plan complete?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Assuming Swarm backs up volumes or databases.
  • Backing up only the swarm directory and losing the autolock key.
  • Running two managers (no fault tolerance; losing either loses quorum).
  • Never testing the restore.
  • Using --force-new-cluster while the old managers are still running, which can split the cluster.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How many managers can fail in a 5-manager swarm while keeping quorum?

    The majority of 5 is 3, so 2 failures are tolerated.
  2. 2. What does docker swarm init --force-new-cluster do?

    It is the standard way to recover when quorum is lost.
  3. 3. Which data is NOT included in a /var/lib/docker/swarm backup?

    Volumes and databases need their own backups.
  4. 4. What happens to running tasks on workers when managers lose quorum?

    Workers continue serving, but scheduling and updates stop until quorum returns.

Summary

  • Back up /var/lib/docker/swarm from one manager at a time, keep definitions in Git, secret originals in a vault and data in database-native backups.
  • Quorum is a majority of managers; use 3 or 5 across failure domains.
  • Recover with --force-new-cluster, then re-add managers and verify.
  • Rehearse restores; a tested plan is the only real plan.

Next: Swarm vs Kubernetes and Other Options