Skip to content
Tutorials Center
My learning

Topics

Lesson 95 of 111 ยท Orchestration Basics

Logging and Monitoring

How Docker collects container logs, logging drivers, json-file rotation, centralized logging, and basic metrics with docker stats, events, cAdvisor and Prometheus.

A container that runs in production must be observable. When it misbehaves at 3 a.m., you need two things: its logs (what the application said) and its metrics (how much CPU, memory and traffic it used over time). Docker gives you a simple, uniform way to capture the first and basic tools for the second, and a few well-known pitfalls, such as the log file that quietly fills a disk, are easy to avoid once you know them.

In this lesson you will see how Docker captures output, how to rotate logs, how logging drivers send them elsewhere, what "centralized logging" means in practice, and how to watch resource usage with docker stats, docker events and the cAdvisor plus Prometheus pair. Commands and output come from Docker 29 unless a block is marked as a read-only example.

How Docker logs work

Docker has one logging rule: the main process of the container should write to standard output and standard error. The daemon attaches to those two streams of PID 1, wraps every line with a timestamp and the stream name, and hands it to a logging driver. The default driver, json-file, writes one JSON object per line to a file on the host. docker logs simply reads it back.

TEXT
 app (PID 1) --stdout/stderr--> containerd-shim --> dockerd --> logging driver
                                                                   |
                          json-file / local   -> file on the host  (docker logs reads it)
                          syslog / journald   -> the host logger
                          fluentd / gelf / splunk / awslogs -> a remote service

That is why the standard advice is "log to stdout, not to a file inside the container". A log file in the writable layer disappears with the container, cannot be read by docker logs, and grows without limit. Some official images solve this with a symlink: the nginx image links /var/log/nginx/access.log to /dev/stdout, and the error log to /dev/stderr.

You can check which driver the daemon uses by default:

CONSOLE
$ docker info --format 'driver={{.LoggingDriver}}'
driver=json-file

Reading logs

Let us start a container that writes to both streams for a few seconds:

CONSOLE
$ docker run -d --name chat alpine:3 sh -c 'i=0; while [ $i -lt 5 ]; do echo "line $i"; echo "oops $i" >&2; i=$((i+1)); sleep 1; done; sleep 300'
3b2aefce1954754d17daf890a81e6719a797982a294e426be3f80cd03313192d
$ docker logs chat
line 0
oops 0
oops 1
line 1
line 2
oops 2
line 3
oops 3
line 4
oops 4

The order of lines from different streams can differ slightly from run to run, because stdout and stderr are separate pipes. docker logs keeps them separate on purpose: it writes the container's stdout to its own stdout and the container's stderr to its own stderr, so you can filter them with ordinary shell redirection:

CONSOLE
$ docker logs chat 2>/dev/null
line 0
line 1
line 2
line 3
line 4
$ docker logs chat 2>&1 >/dev/null
oops 0
oops 1
oops 2
oops 3
oops 4

The options you will use most are -f (follow), -n / --tail (last lines only), -t (timestamps), and --since / --until (a time or a duration such as 10m):

CONSOLE
$ docker logs --tail 2 chat
line 4
oops 4
$ docker logs -t --since 3s chat
2026-10-23T15:17:22.077714280Z oops 4
2026-10-23T15:17:22.077676275Z line 4

(Timestamps are the daemon's, in UTC, with nanoseconds.) More on the day-to-day commands is in Logs and exec. With Compose, docker compose logs merges all services, adds a coloured prefix and also supports --tail, --since, -f and --no-log-prefix:

CONSOLE
$ docker compose logs --no-color --tail 3
web-1     | 2026/10/23 15:20:58 [notice] 1#1: start worker process 33
web-1     | 2026/10/23 15:20:58 [notice] 1#1: start worker process 34
web-1     | 2026/10/23 15:20:58 [notice] 1#1: start worker process 35
worker-1  | tick 4
worker-1  | tick 5
worker-1  | tick 6
$ docker compose logs --no-log-prefix --tail 2 worker
tick 5
tick 6

What is on disk

With json-file, each log line is stored like this (a read-only example of the file's format; the file itself lives inside the Docker host, at the path docker inspect --format '{{.LogPath}}' prints, for example /var/lib/docker/containers/<id>/<id>-json.log):

JSON
{"log":"line 0\n","stream":"stdout","time":"2026-10-23T15:17:17.012345678Z"}
{"log":"oops 0\n","stream":"stderr","time":"2026-10-23T15:17:17.012456789Z"}

Every line carries about 70 bytes of JSON overhead, and the file has no size limit by default. That is the most common logging mistake in Docker: a chatty service writes for months and one day no space left on device appears (see Common Docker Errors and Cleaning Up).

Rotating logs

The json-file and local drivers can rotate files. Two options do the work: max-size (rotate when the file reaches this size) and max-file (how many files to keep, including the current one). You can set them per container to see the effect. This container prints 200 lines of about 55 bytes and is limited to 1 KB per file, three files:

CONSOLE
$ docker run -d --name limited --log-driver json-file \
    --log-opt max-size=1k --log-opt max-file=3 alpine:3 \
    sh -c 'i=0; while [ $i -lt 200 ]; do echo "message number $i with some padding to fill the file"; i=$((i+1)); done; sleep 300'
$ docker inspect --format '{{json .HostConfig.LogConfig}}' limited
{"Type":"json-file","Config":{"max-file":"3","max-size":"1k"}}
$ docker logs limited | wc -l
20
$ docker logs limited | head -2
message number 180 with some padding to fill the file
message number 181 with some padding to fill the file

Only the newest 20 lines survived: the rest were rotated away. Real settings are larger, of course: 10m and 3 is a common choice. To make that the default for every container, set it once in the daemon configuration (/etc/docker/daemon.json on Linux, or Settings > Docker Engine in Docker Desktop), then restart the daemon (a read-only example):

JSON
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

The default only applies to containers created after the change; existing containers keep the logging configuration they were created with. In Compose, set it per service, which is easy to share with the YAML anchor from Multiple Files, Profiles and Extends. This file was checked with docker compose config:

YAML
x-logging: &default-logging
  driver: json-file
  options:
    max-size: "10m"
    max-file: "3"

services:
  web:
    image: nginx:alpine
    logging: *default-logging
  worker:
    image: alpine:3
    command: ["sh", "-c", "while true; do echo tick; sleep 5; done"]
    logging: *default-logging

The local driver is a good alternative to json-file. It stores logs in a compact binary format, rotates by default (20 MB per file, 5 files) and is faster to write. Its one downside is that other tools cannot read the files directly, because the format is internal to Docker. docker logs works as usual.

Logging drivers

A driver decides where the lines go. Ask your daemon which ones it has:

CONSOLE
$ docker info --format '{{json .Plugins.Log}}'
["awslogs","fluentd","gcplogs","gelf","journald","json-file","local","splunk","syslog"]
Driver Destination Good for
json-file JSON file on the host (default) Small setups, tools that read the files
local Compact rotated files on the host Single hosts, with sensible defaults
journald systemd journal Hosts where you already use journalctl
syslog A local or remote syslog server Traditional log infrastructure
fluentd, gelf, splunk, awslogs, gcplogs A collector or cloud service Centralized logging
none Nowhere Containers whose output must not be stored

You choose a driver with --log-driver and give it settings with --log-opt. Two behaviors are worth seeing. First, none means exactly that, and docker logs has nothing to read:

CONSOLE
$ docker run -d --name quiet --log-driver none alpine:3 sh -c 'echo hi-none; sleep 300'
$ docker logs quiet
Error response from daemon: configured logging driver does not support reading

Second, remote drivers can fail at start-up. This fluentd container points at an address where nothing listens, and Docker refuses to create it, because by default the driver connects synchronously:

CONSOLE
$ docker run -d --log-driver fluentd --log-opt fluentd-address=127.0.0.1:18149 alpine:3 echo x
docker: Error response from daemon: failed to initialize logging driver: dial tcp 127.0.0.1:18149: connect: connection refused

Run 'docker run --help' for more information

With --log-opt fluentd-async=true the same container starts, and the driver reconnects in the background. Remember that trade-off for every network driver: strict delivery guarantees block your application when the log server is down, while asynchronous modes may drop lines. Which one you want is a business decision.

Third, a pleasant detail: since Docker 20.10 the daemon keeps a small local copy of the logs even when the driver is a remote one, so docker logs still works. With gelf (UDP) a container that printed x can be read back:

CONSOLE
$ docker run -d --name remote --log-driver gelf --log-opt gelf-address=udp://127.0.0.1:18149 alpine:3 echo x
$ docker logs remote
x

This "dual logging" copy is limited in size and is not a substitute for a real log store.

You can also tag lines and attach labels so the collector can tell services apart. The tag template and the labels are stored in the container's log configuration:

CONSOLE
$ docker run -d --name tagged --log-opt tag="{{.Name}}/{{.ImageName}}" \
    --log-opt labels=app --label app=demo alpine:3 echo tagged
$ docker inspect --format '{{json .HostConfig.LogConfig}}' tagged
{"Type":"json-file","Config":{"labels":"app","tag":"{{.Name}}/{{.ImageName}}"}}

Centralized logging

On one machine, docker logs is enough. With ten machines, or containers that are replaced all the time, it is not: when a container is removed its json file is gone, and you do not want to ssh into servers to grep. Centralized logging moves every log line to one searchable place. There are two common designs:

  1. Driver to collector. Containers use a driver such as fluentd, gelf or awslogs, which sends lines straight to Fluentd, Logstash, Graylog or a cloud service. Simple, but every container needs the setting, and delivery problems affect the application.
  2. Agent reading the host files. Containers keep the default json-file (with rotation), and a collector agent running on every host (Fluent Bit, Vector, Promtail, Filebeat, the OpenTelemetry Collector) tails the files under /var/lib/docker/containers, adds labels such as container name and image, and ships them. The application never knows. This is the standard design on Kubernetes and works equally well with Docker.

The second design in Compose looks like this (a read-only example; the collector's image and configuration depend on the tool you pick):

YAML
services:
  app:
    image: myorg/app:1.4.2
    logging:
      driver: json-file
      options: { max-size: "10m", max-file: "3" }
  log-agent:
    image: fluent/fluent-bit:latest
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro

Whichever you choose, log structured data if you can: one JSON object per line ({"level":"error","msg":"payment failed","order":4411}) is far easier to search than free text. Never log secrets or personal data, and give every request an ID that appears in every line it touches. Application-level logging is the part Docker cannot do for you.

Basic monitoring with Docker itself

Logs tell you what happened; metrics tell you how the system feels. The built-in view is docker stats:

CONSOLE
$ docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}' chat limited
NAME      CPU %     MEM USAGE / LIMIT   PIDS
chat      0.00%     352KiB / 11.63GiB   1
limited   0.00%     364KiB / 11.63GiB   1

The limit column shows the memory available to the container, which here is the whole VM because no --memory was set (Resource Limits). Without --no-stream the table refreshes live. The numbers come from the container's cgroup, so they are cheap to collect. A few things to know: the CPU percentage can exceed 100% on multi-core hosts (200% means two cores busy), and "MEM USAGE" includes some cache, which makes it look higher than the application's own accounting.

docker events is the other built-in stream. It reports what happens to Docker objects: start, stop, die, OOM kill, health status changes, image pulls. Pipe it into a script and you have a basic alert source:

CONSOLE
$ docker events --filter container=chat --format '{{.Action}}'
exec_create: true
exec_start: true
exec_die
kill
pause
unpause

Those lines appeared while running docker exec chat true, docker kill -s SIGUSR1 chat, and docker pause followed by docker unpause. Events like oom and die (with an exit code) are exactly what you want to alert on. Add --since and --until to look at the past.

cAdvisor and Prometheus

For history, graphs and alerts you need a monitoring system. The classic open-source combination has three parts:

  • cAdvisor (from Google) reads cgroup data from the host and exposes metrics per container on an HTTP endpoint.
  • Prometheus scrapes those endpoints on a schedule (every 15 seconds, say) and stores the time series.
  • Grafana draws dashboards and Alertmanager sends the alerts.

A minimal Compose file to try them on one host looks like this. It is a read-only example: cAdvisor needs read access to host paths (which is why it must be run only on a host you control), and it was only checked for syntax with docker compose config:

YAML
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    ports:
      - "127.0.0.1:8081:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "127.0.0.1:9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro

with a prometheus.yml that tells Prometheus where to scrape:

YAML
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: cadvisor
    static_configs:
      - targets: ["cadvisor:8080"]

Prometheus then answers questions such as "which container used the most memory in the last hour?". The metric names come from cAdvisor, for example container_memory_usage_bytes and container_cpu_usage_seconds_total, and you can query them in Prometheus's web UI on port 9090. Pin real versions instead of latest for anything that stays up.

Alternatives are plenty: Datadog, New Relic and Grafana Cloud agents, the OpenTelemetry Collector, and on Kubernetes the Prometheus Operator. What matters is the habit, not the product: collect container CPU, memory, restarts and OOM kills, plus application metrics such as request rate, error rate and latency exposed by the app itself.

What to alert on

  • A container restarting more than a few times per hour (crash loops).
  • Memory close to the limit, or any OOM kill.
  • Healthcheck status unhealthy (HEALTHCHECK).
  • Disk usage of /var/lib/docker.
  • Error rate and latency of the application itself, which matter more than any container metric.

Clean up your logging experiments

Containers keep their logs until they are removed. When you are done:

Terminal
docker rm -f chat limited quiet remote tagged

Removing a container deletes its log files too, which is one reason centralized logging matters.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What do you get from docker logs on a container started with --log-driver none?

    With none, Docker stores nothing, so docker logs refuses with: configured logging driver does not support reading.
  2. 2. Why can a container with default settings eventually fill the host disk with logs?

    By default json-file keeps one file that grows without a limit. Set max-size and max-file per container, in Compose, or globally in daemon.json.
  3. 3. You change log-opts in daemon.json and restart the daemon. Which containers use the new rotation settings?

    A container's logging configuration is fixed when it is created. Existing containers must be recreated to pick up new defaults.
  4. 4. Which is the usual design on Kubernetes and with agent-based logging on plain Docker?

    Containers keep writing to stdout/stderr, and an agent such as Fluent Bit or Vector reads the host files, adds metadata and forwards them.
  5. 5. In docker stats, what can a CPU % of 200% mean?

    CPU percentage is relative to one core, so a busy multi-threaded process can exceed 100%.

Summary

  • Containers should write logs to stdout and stderr; the daemon captures them and passes them to a logging driver, and docker logs reads them back (stdout and stderr stay separate).
  • The default json-file driver never rotates unless you set max-size and max-file; set them per container, in Compose or in daemon.json (only new containers pick up daemon defaults). The local driver rotates by default.
  • Drivers such as syslog, journald, fluentd, gelf and awslogs ship logs elsewhere; none stores nothing. Synchronous remote drivers can stop a container from starting when the server is down.
  • Centralized logging usually means an agent such as Fluent Bit or Vector reading the host's log files, or a network driver; structured one-line JSON logs make searching easy.
  • docker stats and docker events give live resource usage and lifecycle events; cAdvisor plus Prometheus (and Grafana) add history, graphs and alerts.
  • Alert on restarts, OOM kills, unhealthy status, disk usage and the application's own error rate and latency.

Next: Docker in Docker and Socket Access