Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 38 ยท Services and Tasks

Health Checks and Restart Policies

Teach Swarm to detect broken containers with health checks, control restarts with restart conditions, delays and windows, and avoid crash loops.

A container that is running is not necessarily working. A web server can deadlock, a database connection pool can break, and the process still exists. Swarm detects process exits on its own, but to notice "running but broken" you must provide a health check. And to decide what happens afterwards you configure a restart policy. Together they are the self-healing engine.

Health checks

A health check is a command that runs inside the container at an interval. Exit code 0 means healthy, 1 means unhealthy.

CONSOLE
$ docker service create --name web \
    --health-cmd 'curl -fsS http://localhost/ || exit 1' \
    --health-interval 15s \
    --health-timeout 3s \
    --health-retries 3 \
    --health-start-period 20s \
    nginx:1.27
Flag Meaning
--health-cmd Command to run (shell form; use CMD-SHELL semantics).
--health-interval Time between checks (default 30s).
--health-timeout A check taking longer counts as a failure (default 30s).
--health-retries Consecutive failures before the container is unhealthy (default 3).
--health-start-period Grace time at startup during which failures are not counted.
--no-healthcheck Disable a check baked into the image.

You can also define HEALTHCHECK in the Dockerfile; service flags override it. The image must contain the tool the command uses (curl, wget, pg_isready): a missing tool makes every check fail. Prefer a lightweight endpoint such as /healthz that verifies what the service needs without heavy work.

What Swarm does with the result

  • A task is starting until the first successful check (or the start period ends), then running.
  • If the container becomes unhealthy, Swarm treats the task as failed, shuts it down and creates a replacement according to the restart policy.
  • During rolling updates, Swarm waits for a new task to be healthy before continuing, and for services with ingress ports, the routing mesh sends traffic only to healthy tasks. Without a health check, "running" is the only signal, so traffic may hit a container before the app is ready.

Restart policies

CONSOLE
$ docker service create --name worker \
    --restart-condition on-failure \
    --restart-delay 10s \
    --restart-max-attempts 5 \
    --restart-window 2m \
    myorg/worker:1
Flag Meaning
--restart-condition any (default): always replace; on-failure: only after a non-zero exit or health failure; none: never.
--restart-delay Wait before replacing (default 5s).
--restart-max-attempts Give up after this many restarts (default: unlimited).
--restart-window Time window in which attempts are counted; attempts that happen outside the window do not count.

Use any for long-running services, on-failure for jobs that should end cleanly with code 0 (so a successful finish is not restarted), and none for one-shot tasks you want to inspect.

Example of the window: with max-attempts 3 and window 1m, a task that crashes four times within a minute stops being restarted; a task that crashes once an hour never reaches the limit.

Remember from the task lifecycle: a "restart" creates a new task, possibly on another node, so a node-specific problem (a bad disk) does not trap the service forever.

Crash loops

A container that fails at start produces a fast loop: start, crash, delay, start. Symptoms are a growing list in docker service ps, and logs ending abruptly. Defences:

  1. Keep --restart-delay at several seconds so you do not hammer dependencies.
  2. Set --restart-max-attempts for batch work.
  3. Use --update-failure-action rollback so a bad release returns to the previous version (updates).
  4. Read docker service logs --tail 50 name first; the cause is usually a missing environment variable or an unreachable database.

Stop behaviour

When a task is shut down (scale down, update), Docker sends SIGTERM (or --stop-signal) and waits --stop-grace-period (default 10s) before SIGKILL. Make your application handle SIGTERM for graceful shutdown, and run it as PID 1 (or with --init) so that it actually receives the signal.

Stand-in: health check state machine

The simulator below feeds a series of check results through the rule "N consecutive failures after the start period mean unhealthy" and shows the effect on the task.

Python Stand-in: health check evaluation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: restart window

This stand-in applies --restart-max-attempts and --restart-window to a list of crash times (in seconds) to show when Swarm would give up.

Python Stand-in: restart attempts within a window

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. What does Swarm do with a task whose container becomes unhealthy?

    An unhealthy container fails the task, and the orchestrator replaces it.
  2. 2. Which restart condition suits a batch job that should not be restarted after it exits with code 0?

    on-failure restarts only after errors. 'always' and 'unless-stopped' are docker run policies, not service conditions.
  3. 3. What is --health-start-period for?

    Slow-starting applications need time before health failures should count.

Summary

  • Health checks detect running-but-broken containers; unhealthy means a failed task.
  • Restart conditions are any, on-failure and none, with delay, attempts and window.
  • A restart is a new task, not the same container.
  • Handle SIGTERM and tune the stop grace period.

Next: Overlay Networks