Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 37 ยท Containers and Images

Health Checks and Restart Policies in Podman

Make containers self-monitoring: add health checks with intervals and retries, react to failures, set restart policies, and restore containers after a reboot.

A container that is "running" is not necessarily a container that is "working". The process may be alive but stuck, out of database connections, or answering every request with an error. Health checks let a container report whether it is really OK, and restart policies decide what happens when it stops. Together they are the basic reliability tools for a single machine.

This lesson shows how to define a health check, read its status, react to failures, choose a restart policy, and the important caveat that makes Podman different: after a reboot, nothing restarts containers unless you set that up.

A first health check

A health check is a command that Podman runs inside the container at regular intervals. Exit code 0 means healthy, any other code means unhealthy.

Terminal
podman run -d --name web \
  --health-cmd 'wget -q --spider http://127.0.0.1/ || exit 1' \
  --health-interval 10s \
  --health-timeout 3s \
  --health-retries 3 \
  --health-start-period 5s \
  -p 8080:80 docker.io/library/nginx:alpine
Option Meaning Typical value
--health-cmd the command to run (shell form or JSON array) a request or a small script
--health-interval time between checks 30s (the default)
--health-timeout how long one check may take 5s
--health-retries consecutive failures before "unhealthy" 3
--health-start-period grace time while the app boots; failures do not count 10s to 60s
--health-on-failure action after becoming unhealthy none, kill, restart, stop

The command runs inside the container, so the tool it uses (wget, curl, pg_isready) must exist in the image. Keep the check cheap: it runs often, and a heavy check can itself cause the problem.

Reading the status

Terminal
podman ps                                   # the status column shows (healthy), (unhealthy) or (starting)
podman healthcheck run web                  # run the check now; exit code 0 means healthy
podman inspect --format '{{.State.Health.Status}}' web
podman inspect --format '{{json .State.Health.Log}}' web

The state moves through starting, then healthy or unhealthy. The log keeps the last few results with their output and exit code, which is the first place to look when a check fails. Podman implements the interval with a systemd timer, so rootless setups need a working user systemd session; without systemd, scheduled checks may not run, and you can still call podman healthcheck run yourself.

Reacting to failure

By default an unhealthy container only gets a label: nothing else happens. You choose a reaction with --health-on-failure:

  • none (default): just record it.
  • kill: kill the container, so the restart policy can bring it back.
  • restart: restart the container.
  • stop: stop it.
Terminal
podman run -d --name api --restart on-failure \
  --health-cmd '/app/ping' --health-retries 2 --health-interval 15s \
  --health-on-failure kill my-registry/api:1.0

With kill plus --restart on-failure, a stuck application is killed with a non-zero exit and restarted. Choose deliberately: restarting a container that fails because the database is down can create a restart storm, so combine it with a sensible --health-start-period and retry count.

Here is a stand-in that simulates health states with retries and a start period. It is JavaScript logic, not real health checking.

JS Stand-in: health state machine

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that the two failures in the start period do not count, one success makes the container healthy again, and only three failures in a row make it unhealthy.

Health checks in a Containerfile

Images can carry their own check with the HEALTHCHECK instruction:

DOCKERFILE
FROM docker.io/library/nginx:alpine
HEALTHCHECK --interval=15s --timeout=3s --retries=3 CMD wget -q --spider http://127.0.0.1/ || exit 1

There is a catch: the OCI image format has no field for health checks. Podman builds OCI images by default, so the instruction is ignored with a warning. Build with --format docker to keep it:

Terminal
podman build --format docker -t localhost/web-health .

You can always override or disable at run time: --health-cmd none removes the check.

Restart policies

Terminal
podman run -d --restart no ...            # default: never restart
podman run -d --restart on-failure:5 ...  # restart if exit code is non-zero, at most 5 times
podman run -d --restart always ...        # restart whenever it exits, except when you stop it yourself
podman run -d --restart unless-stopped ...
Policy Restarts after crash Restarts after podman stop Restarts after reboot (with the service enabled)
no no no no
on-failure[:N] yes (non-zero exit) no no
always yes no yes
unless-stopped yes no yes, unless it was stopped manually

Two things to keep straight. First, the restart on a crash is done by Podman's runtime monitoring, using the same mechanism as the health actions. Second, restart after a boot is not done by Podman. When the machine starts, nothing is running. Podman ships a systemd service that starts containers whose policy is always:

Terminal
# rootful containers
sudo systemctl enable --now podman-restart.service

# rootless containers, for your user
systemctl --user enable --now podman-restart.service
loginctl enable-linger "$USER"      # start the user's services at boot, without a login

loginctl enable-linger is the piece people forget: without it, your user's systemd instance (and everything under it) starts only when you log in and stops when the last session ends. For production services, however, the better tool is a systemd unit per container, which gives you ordering, logs and dependencies. Later chapters show how to write those as Quadlet files.

Choosing what to do

  • Development: --restart no and read the logs.
  • A job that may fail transiently: --restart on-failure:3.
  • A long-running service on a server: a systemd/Quadlet unit with Restart=always (or on-failure), plus a health check.
  • Do not rely on --restart always alone for anything important.

Here is a small decision helper as a stand-in, which also shows how restart counting works.

JS Stand-in: restart policy decisions

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does exit code 0 from a health check command mean?

    Exit code 0 means success (healthy). Any other code counts as a failure.
  2. 2. What is the purpose of --health-start-period?

    Slow-starting programs would otherwise be marked unhealthy before they are ready.
  3. 3. Why might a HEALTHCHECK in your Containerfile be ignored?

    podman build produces OCI images by default, which cannot store HEALTHCHECK. The docker format can.
  4. 4. After a host reboot, what restarts containers with --restart always?

    Podman has no daemon. A systemd service (podman-restart.service) or per-container units start them at boot.
  5. 5. Rootless user services should start at boot without login. Which command enables this?

    Linger keeps the user's systemd instance running, so user services start at boot and survive logout.

Summary

  • Health checks run a command in the container at intervals; exit 0 is healthy.
  • Tune interval, timeout, retries and start period, and read results with podman healthcheck run and inspect.
  • --health-on-failure can kill, restart or stop an unhealthy container; by default it only records.
  • Containerfile HEALTHCHECK needs --format docker, because OCI images cannot store it.
  • Restart policies handle crashes; reboots need podman-restart.service or systemd/Quadlet units, plus lingering for rootless users.

Next: Rootless Podman and User Namespaces