Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Operations

Consul Backups, Upgrades and Troubleshooting

Operate Consul safely: take and restore snapshots, plan rolling upgrades with consul leave, handle failed servers and follow a checklist to troubleshoot common problems.

Three routine jobs keep a Consul cluster alive for years: backups you have actually tested, upgrades that never break quorum, and a calm approach to troubleshooting when something misbehaves. This lesson gives you a procedure for each.

The product cannot run on this site. Commands are read-only and the runnable examples are stand-in checkers and planners.

Snapshots

A snapshot is an atomic, point-in-time archive of the server state: catalog, KV store, sessions, ACLs, config entries and prepared queries. It does not contain agent configuration files, certificates or the gossip key; keep those in your own backup.

Terminal
consul snapshot save backup.snap
consul snapshot inspect backup.snap
consul snapshot restore backup.snap

save asks the leader (or any server with -stale) for the state, and needs a token with acl = "write" when ACLs are enabled. The file is not encrypted by Consul, and it contains secrets such as ACL token secrets and KV values. Encrypt it and restrict access before copying it to object storage.

A good schedule: hourly or daily snapshots, copies in a different account or region, retention of weeks, and a monthly restore drill into a scratch cluster. A backup that has never been restored is an assumption, not a backup.

Consul Enterprise has an automated snapshot agent (consul snapshot agent); in open source, run consul snapshot save from cron or a pipeline and upload the file.

Python Stand-in: backup retention policy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Restoring

Restoring replaces the entire state of the cluster with the snapshot, so use it deliberately: after data corruption, an accidental mass deletion or to build a copy.

  1. Prefer restoring into a new cluster to check the data first.
  2. On the target, run consul snapshot restore backup.snap against the leader (needs an ACL token if enabled). It replicates to all servers.
  3. Agents will re-register their own services, so the catalog quickly corrects itself; KV, ACLs and config entries come back from the snapshot.
  4. With ACLs on, tokens in the snapshot are valid again, including the bootstrap token as it was at backup time.

Restoring to a cluster with a different datacenter name or Raft history is a migration exercise; test it.

Rolling upgrades

Upgrade servers one at a time, keeping quorum. The general procedure:

  1. Read the release notes and the upgrade guide for every version you skip. Upgrade one minor version at a time (for example 1.20 to 1.21), and review breaking changes.
  2. Take a snapshot.
  3. Upgrade followers first, one by one. For each: stop the service (the agent runs consul leave on a graceful shutdown), replace the binary, start it, and wait until consul operator raft list-peers and consul operator autopilot state show it healthy before moving on.
  4. Upgrade the leader last. Stopping it triggers an election; with consul leave handing off cleanly, impact is brief.
  5. Then upgrade client agents, in batches that do not remove every instance of a service at once.
  6. Verify: members all alive, services healthy, a test lookup and a test write.

The agent configuration leave_on_terminate and skip_leave_on_interrupt decide whether stopping an agent is a graceful leave. For servers, a graceful leave removes the server from the Raft peer set; if you only restart, you do not want it removed. Because of that, servers use leave_on_terminate = false by default and clients true.

Compatible versions matter: during the rolling window, mixed versions must interoperate. Do not leave a cluster half-upgraded for days, and never downgrade across state-format changes; restore from snapshot instead.

Python Stand-in: rolling upgrade planner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A troubleshooting method

When something is wrong, work from the outside in and avoid guessing.

  1. Is there a leader? curl http://127.0.0.1:8500/v1/status/leader. If empty, focus on quorum, the network between servers and recent restarts.
  2. Who is in the cluster? consul members, consul members -wan, consul operator raft list-peers. Compare with what you expect.
  3. What do the logs say? Read the last 15 minutes of server logs, or consul monitor -log-level=debug on the affected agent.
  4. Is it permissions? ACL not found and Permission denied mean the token or policy is wrong, not the cluster. Check with consul acl token read -self using the same token.
  5. Is it time or certificates? TLS errors usually mean expired certificates or clock skew. Check date on all nodes and the validity dates.
  6. Is it the network? Ports 8300, 8301 (TCP and UDP), 8302 for WAN, and 8600 for DNS, plus firewalls, security groups and MTU.
  7. Is it load or disk? High disk latency or a full disk breaks Raft. Check metrics from the last lesson.

Common problems and fixes

Symptom Likely cause What to do
No cluster leader Lost quorum, network partition, disk stalls Restore enough servers; check connectivity; Raft recovery as last resort
Agent will not join Wrong gossip key, firewall, unreachable server Check encrypt, ports and retry_join
Service shows critical Health check failing Run the check manually; see the Health Checks chapter
Permission denied Token lacks a rule Review the token's policies; ACL cache TTL
DNS returns nothing Anonymous token lacks service/node read, or no healthy instance Check ACLs and ?passing filters
Flapping nodes Overloaded agents, packet loss, short check timeouts Investigate CPU and network; tune serf_lan timings only after finding the cause
Sidecar 503 Intention deny, no healthy upstream, certificate trouble consul intention check, consul catalog services, proxy admin /clusters
Cluster slow Slow disks, huge KV values, excessive watches or blocking queries Check commit time; reduce client polling
Python Stand-in: symptom to next step lookup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Bringing a failed server back or replacing it

  • Same node, same data: just restart it; it replays the Raft log and catches up.
  • Lost data directory: start a new server with the same name and a new node ID. Autopilot cleans up the old peer, or remove it with consul operator raft remove-peer -id <id>.
  • Permanently gone: add a fresh server first (so you keep quorum), wait for it to be promoted, then remove the dead one.

Never copy a server's data directory to another server; node IDs would collide and the Raft state would be corrupt.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a Consul snapshot NOT contain?

    Snapshots hold server state. Keep configuration, certificates and the gossip key in your own backups.
  2. 2. In which order should servers be upgraded?

    This keeps quorum and causes a single, brief leader election at the end.
  3. 3. Why test restoring a snapshot regularly?

    Drills prove the backup is usable and the procedure is understood before a crisis.
  4. 4. You see Permission denied in a client's logs. What should you check first?

    Permission errors point to ACLs, not to cluster health.
  5. 5. What is the safe way to replace a dead server?

    Never duplicate node IDs or data directories; add the replacement first to keep quorum.

Summary

  • Save snapshots on a schedule, encrypt them, store them off-site and rehearse restores; back up configs and keys separately.
  • Upgrade one minor version at a time, followers first and the leader last, checking Raft and Autopilot between steps.
  • Troubleshoot outside in: leader, membership, logs, ACLs, time and certificates, network, load.
  • Replace failed servers by adding first and removing second, with a new node ID.

Next: Project: A Service Registry Simulator