Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 38 ยท Operations and Alternatives

Monitoring and Logging a Docker Swarm

See what is happening in a swarm: docker service logs, events, node and task states, logging drivers, a global log-agent pattern and metrics with Prometheus-style tools.

A cluster that nobody watches fails silently. Swarm gives you built-in commands for quick answers and leaves long-term metrics and log storage to the tools you attach. This lesson covers both layers: what to ask Docker directly, and how to ship logs and metrics somewhere durable.

Swarm is not available on this site, so commands are read-only text and I do not print sample output I cannot verify. The Python blocks are stand-ins that parse and aggregate the kinds of data you will meet.

Quick answers from the CLI

Terminal
docker node ls                         # node status and availability
docker service ls                      # replicas desired versus running
docker service ps api --no-trunc       # task history and errors
docker service logs -f --tail 100 api  # logs from all tasks of a service
docker service logs --since 10m --timestamps api
docker events --filter type=service    # live stream of changes
docker system df                       # disk used by images, volumes, cache

docker service logs merges the log streams of all replicas and prefixes each line with the task name, which is how you spot "only replica 3 is failing". It works only with logging drivers that can be read back (json-file, local, journald). With drivers that ship logs away (such as gelf or fluentd) the command has nothing to show.

A short daily check: all nodes Ready and Active, one manager Leader and the others Reachable, services at N/N, no tasks looping in Failed/Rejected, free disk on every node.

Logging drivers

By default containers use json-file, which grows without limit unless you cap it. Set rotation in /etc/docker/daemon.json on every node:

JSON
{
  "log-driver": "local",
  "log-opts": { "max-size": "10m", "max-file": "5" }
}

Or per service in the stack file:

YAML
services:
  api:
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Unbounded logs are one of the most common reasons nodes fill their disks.

Central logging

Applications should log to stdout and stderr and not to files. Then a collector moves the logs to a central store (Loki, Elasticsearch, a cloud service). Two patterns work well in Swarm:

  1. Logging driver: each container sends logs straight to the collector (for example gelf or fluentd). Simple, but if the collector is down, container start or log delivery can be affected unless you configure non-blocking mode (mode: non-blocking).
  2. Global agent: run the collector as a mode: global service that reads container log files, so every node has exactly one agent (the deploy key).
YAML
services:
  log-agent:
    image: myorg/log-agent:2.0
    deploy:
      mode: global
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro

Metrics

Three layers of metrics matter:

  • Host: CPU, memory, disk and network per node (a node exporter as a global service).
  • Containers: per-task usage (cAdvisor as a global service, or the Docker engine's own metrics endpoint, enabled with "metrics-addr": "127.0.0.1:9323" in daemon.json).
  • Application: request rate, latency, errors, queue depth, exposed by your code.

A common stack is Prometheus (scrapes), Grafana (dashboards) and Alertmanager (pages), deployed as a Swarm stack. Because Swarm DNS provides tasks.<service> names that return every task's IP, Prometheus can discover all replicas of a global exporter with a DNS-based service discovery rule. Add a service and node label to every metric so a graph can be sliced by either.

flowchart LR
  A["Containers<br/>(stdout)"] --> L["Log agent<br/>(global)"] --> S["Central log store"]
  E["Exporters<br/>(global)"] --> P["Prometheus"] --> G["Grafana + alerts"]

Logs and metrics each travel through one agent per node.

What to alert on

  • A node is Down or a manager loses quorum (more than half of managers unreachable).
  • A service has fewer running tasks than desired for more than a few minutes.
  • Disk or memory on a node above 85%.
  • Restarts: a task that restarts more than a few times in ten minutes.
  • User-facing symptoms: error rate and latency, which matter more than any internal metric.

Stand-in: spotting a failing replica

Python Stand-in: group service log lines by task

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: a small alert evaluator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Leaving the default json-file driver without max-size.
  • Writing logs to files inside containers where nobody can collect them.
  • Monitoring containers but not the nodes beneath them.
  • Alerting on every restart instead of on user impact and sustained conditions.
  • Running a log or metrics stack on a single node, so it disappears exactly when the node fails.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. Which command streams the combined logs of all tasks of a service?

    docker service logs aggregates the task logs and prefixes lines with the task name.
  2. 2. Why configure max-size and max-file for logging?

    Without rotation json-file logs grow until the disk is full.
  3. 3. Which deploy mode puts exactly one log-agent task on every node?

    Global services run one task per node.

Summary

  • Use docker node ls, service ls, service ps and service logs for fast diagnosis.
  • Cap log size on every node and send stdout/stderr to a central store via a driver or a global agent.
  • Collect host, container and application metrics and alert on sustained, user-visible conditions.
  • Keep your monitoring stack off single points of failure.

Next: Backup and Disaster Recovery