Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Systemd and Operations

Health Checks, Logs and Events

Monitor Podman containers: define health checks, read health status, choose log drivers, use podman logs, events, stats and top, and send logs to the systemd journal.

"The container is running" and "the application works" are different statements. A web server process can be alive but stuck; a database can be starting for thirty seconds before it accepts connections. Podman gives you three tools to tell the difference: health checks, logs and events, plus live resource numbers from stats and top.

All of these need real containers, so the commands are read-only. The stand-ins model the health state machine and parse log lines.

Health checks

A health check is a command Podman runs inside the container on a schedule. Exit code 0 means healthy; anything else counts as a failure.

Terminal
podman run -d --name web \
  --health-cmd 'curl -fsS http://localhost/ || exit 1' \
  --health-interval 10s \
  --health-timeout 3s \
  --health-retries 3 \
  --health-start-period 20s \
  docker.io/library/nginx:stable
Option Meaning
--health-cmd The command (runs inside the container, so it must exist in the image)
--health-interval Time between checks (default 30s)
--health-timeout A run longer than this counts as failed
--health-retries Consecutive failures before the container is unhealthy
--health-start-period Grace time at startup: failures do not count until it ends, but a success marks the container healthy immediately
--health-on-failure Action when unhealthy: none (default), kill, restart or stop

Read the status:

Terminal
podman ps                                   # shows (healthy), (unhealthy) or (starting)
podman healthcheck run web                  # run the check now; exit code reports result
podman inspect web --format '{{.State.Health.Status}}'
podman inspect web --format '{{json .State.Health}}'

A check must be cheap and meaningful. Test the thing users depend on (an HTTP response, a pg_isready result), not just that a process exists. Remember Podman's default OCI build format has no HEALTHCHECK instruction, so define checks at run time, in Quadlet (HealthCmd=, HealthInterval=, HealthRetries=, HealthOnFailure=) or in Compose, or build with --format docker.

Health states

starting Inside the start period.
healthy Check passes.
unhealthy Too many failures in a row.

Checks run on a timer.

1 / 2

The state machine

Python Health state simulator (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Acting on health

A plain unhealthy label does nothing on its own. Options:

  • --health-on-failure=restart (or kill / stop) for simple self-healing.
  • A systemd unit with --sdnotify=healthy so dependent services wait until the application is genuinely ready, and so podman auto-update rollback can detect a bad update.
  • External monitoring that reads podman inspect or the events stream.

Logs

Container output (stdout and stderr) is captured by a log driver:

Driver Where logs go
journald The systemd journal (default when systemd is the init system)
k8s-file A file in the container's storage directory (default otherwise)
none Discarded
Terminal
podman logs web                    # everything
podman logs -f --tail 50 web       # follow, last 50 lines
podman logs --since 10m web        # last ten minutes
podman logs -t web                 # with timestamps
journalctl --user -u web.service   # logs of a Quadlet service
journalctl --user CONTAINER_NAME=systemd-web

Set the driver per container with --log-driver or globally in containers.conf (log_driver = "journald"). The k8s-file driver can rotate files with --log-opt max-size=10m; with journald, rotation is journald's job (SystemMaxUse= in journald.conf).

Applications should write logs to stdout/stderr, not to files inside the container, so podman logs and the journal see them.

Reading structured lines

Python Filter log lines (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Events

Podman emits an event whenever something happens to a container, image, volume, pod or network:

Terminal
podman events                                   # stream, Ctrl+C to stop
podman events --filter event=die --since 1h
podman events --filter container=web --format '{{.Time}} {{.Status}}'

Typical statuses are create, start, died, health_status, pull, remove. Events are how tools like monitoring agents learn that a container died or became unhealthy. Where events are stored depends on events_logger (journald by default under systemd, otherwise a file).

Live numbers: stats and top

Terminal
podman stats --no-stream            # CPU, memory, network, block I/O once
podman stats web                    # live
podman top web                      # processes inside the container
podman top web user pid args        # choose columns

Memory and CPU limits (--memory 512m, --cpus 1.5) make these numbers meaningful: a container that hits its memory limit is killed (OOM) and shows exit code 137.

Exit code 137 (128 + 9) means the process received SIGKILL, often from the out-of-memory killer. podman inspect --format '{{.State.OOMKilled}}' NAME confirms it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a health check exit code of 0 mean?

    Exit 0 is success; non-zero is a failure that counts toward the retries.
  2. 2. What is --health-start-period for?

    Applications that take time to boot should not be marked unhealthy while starting.
  3. 3. Which command shows the last ten minutes of a container's output?

    podman logs --since limits the output to a recent time window.
  4. 4. Why should applications log to stdout and stderr?

    The log driver reads stdout and stderr; files inside the container are invisible to it.

Summary

  • Health checks run a command on a schedule; interval, timeout, retries and start period define the state machine, and --health-on-failure can act on it.
  • Define checks at run time, in Quadlet or Compose, since OCI images have no HEALTHCHECK.
  • podman logs reads the log driver's output (journald or k8s-file); use --since, -f and --tail.
  • podman events streams lifecycle and health events; stats and top show live usage.
  • Exit code 137 usually means a memory limit kill.

Next: Podman Machine and Remote Podman