Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 38 ยท Scaling and Updates

Health Checks and Rollbacks in Docker Swarm

Add HEALTHCHECK or healthcheck settings so Swarm knows when a task is ready, learn how unhealthy tasks are replaced, and use docker service rollback safely.

A running process is not the same as a working service. A web server can be up while its database connection is broken, and a freshly started container may need ten seconds to warm up. Swarm trusts the container's health check to tell the difference. A good health check makes rolling updates safe, lets Swarm replace sick tasks on its own, and keeps the routing mesh from sending users to containers that cannot answer.

Swarm is not available on this site, so commands and YAML are read-only; the Python blocks are stand-ins that model check results and rollback.

Defining a health check

In a Dockerfile:

DOCKERFILE
HEALTHCHECK --interval=15s --timeout=3s --start-period=20s --retries=3 \
  CMD wget -qO- http://localhost:8080/healthz || exit 1

Or in the stack file, which overrides the image:

YAML
services:
  api:
    image: myorg/api:1.5
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/healthz"]
      interval: 15s
      timeout: 3s
      retries: 3
      start_period: 20s

Docker runs the command inside the container. Exit code 0 means healthy, 1 means unhealthy. The fields:

  • interval: time between checks.
  • timeout: a check slower than this counts as a failure.
  • retries: consecutive failures before the container is marked unhealthy.
  • start_period: a grace time at startup; failures during it do not count (a success ends the period early).

The command must exist in the image. A distroless or scratch image has no wget or curl, so ship a tiny binary or an endpoint-probing flag in your own program (myapp --healthcheck).

What Swarm does with health

  • Rolling updates wait for the new task to become healthy (with start-first, before stopping the old one) and count a failure if it never does. See Rolling Updates.
  • Unhealthy tasks are replaced. When a task's container turns unhealthy, the orchestrator shuts it down and starts a new task, even though the process is still running.
  • The routing mesh only sends traffic to healthy tasks of a service, and DNS round-robin for dnsrr services returns healthy tasks.

Look at the result:

Terminal
docker service ps api --no-trunc
docker inspect --format '{{json .State.Health}}' <container-id>

Run the second command on the node that hosts the container.

Designing a good /healthz

  • Liveness vs readiness. Docker has only one health state, so decide what it means. For most services use "can serve requests": the process is up and essential dependencies answer.
  • Keep it cheap. The check runs every few seconds on every replica. Do not run a heavy query.
  • Be careful with dependencies. If the check fails whenever the database is down, all replicas go unhealthy at once and Swarm churns tasks, which makes a database blip worse. Many teams check only local state, and report dependency problems through metrics instead.
  • No side effects. Health endpoints must not change data.

Rollbacks

Swarm remembers the previous service specification. Roll back manually:

Terminal
docker service rollback api
docker service update --rollback api

Both revert the service to its previous spec (image, config, secrets, resources), using the rollback_config settings:

YAML
deploy:
  rollback_config:
    parallelism: 1
    delay: 0s
    order: stop-first
    failure_action: pause
    monitor: 10s

Only one previous version is kept: rolling back twice returns you to the newer version again. If you need to return several versions, deploy the older tag explicitly from your stack file. Automatic rollback happens when the update's failure_action is rollback, as shown in the previous lesson.

stateDiagram-v2
  [*] --> starting
  starting --> healthy: check passes
  starting --> unhealthy: retries exhausted
  healthy --> unhealthy: retries exhausted
  unhealthy --> replaced: orchestrator starts new task
  replaced --> [*]

An unhealthy task is replaced even if its process is still running.

Stand-in: how interval, retries and start_period interact

Python Stand-in: when does a container become unhealthy?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: a service, a rollback and one level of history

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • No health check at all, so the service is "healthy" the moment the process starts.
  • A start_period shorter than the app's boot time, which causes restart loops.
  • A health check that depends on every downstream system.
  • Using curl in an image that does not contain it (the check then always fails).
  • Expecting docker service rollback to step back more than one version.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A container's process is running, but its health check fails 3 times in a row (retries=3). What does Swarm do?

    Unhealthy tasks are shut down and replaced by new ones.
  2. 2. What is start_period for?

    It gives slow applications time to boot before failures count.
  3. 3. How many previous versions does docker service rollback return to?

    Swarm keeps only the previous service spec.
  4. 4. Why can a health check that tests the database be risky?

    Mass replacement does not fix the database and can worsen the incident.

Summary

  • Health checks tell Swarm whether a task can serve; without one, "started" counts as healthy.
  • Tune interval, timeout, retries and start_period to your application's real behaviour.
  • Unhealthy tasks are replaced, and rolling updates wait on health.
  • docker service rollback returns to the previous spec only; keep older versions in Git for deeper rollbacks.

Next: Resource Limits and Reservations