Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 111 ยท Containers in Depth

Metrics, Tracing and Dashboards for Containers

Build container observability with cAdvisor, Prometheus, Grafana and OpenTelemetry: useful PromQL queries, app metrics, traces and a Compose stack to start from.

Logging and Monitoring got you logs, docker stats and a first Prometheus. That is enough to notice that something is wrong. It is not enough to find out why a request took four seconds yesterday at 14:07, or whether the container that keeps restarting is short of memory or short of CPU. For that you need two more signals, kept over time and shown on a dashboard: metrics and traces.

This lesson builds the common open-source stack step by step: cAdvisor measures every container, Prometheus stores the numbers, Grafana draws them, and OpenTelemetry carries traces from your application to a viewer. You will also learn which queries are worth writing first, and the cardinality mistakes that make monitoring systems fall over.

The stack needs images from several registries and a real Linux host to see container names, and the daemon used to verify this track is a shared Docker Desktop. So the Compose file below was validated with docker compose config (it parses and resolves), but the services were not started. Outputs marked "read-only example" follow the documented formats. Pin the image tags to versions you have checked; the ones here are examples.

Logs, metrics and traces

These three kinds of data answer different questions:

Signal What it is Answers Typical tool
Logs Timestamped lines of text events What exactly happened in this one case? Docker logging drivers, Loki, Elasticsearch
Metrics Numbers sampled over time How much, how often, is it getting worse? Prometheus, Grafana
Traces The path of one request across services, with timing per step Where did this request spend its time? OpenTelemetry, Jaeger, Tempo

Metrics are cheap and tell you that something is off. Traces tell you where. Logs give the detail. The art is moving between them: a metrics alert fires, a trace shows the slow database call, and the logs of that service at that moment explain the error.

flowchart LR
    subgraph host["Docker host"]
        C1["container: app"]
        C2["container: db"]
        CA["cAdvisor"]
        C1 -. cgroup stats .-> CA
        C2 -. cgroup stats .-> CA
    end
    CA -->|"/metrics"| P[("Prometheus")]
    C1 -->|"/metrics (app)"| P
    C1 -->|"OTLP traces"| OC["OTel Collector"]
    OC --> J["Jaeger"]
    P --> G["Grafana"]
    J --> G

Metrics are pulled by Prometheus; traces are pushed to a collector; Grafana can show both.

Metrics: cAdvisor, Prometheus and Grafana

Containers do not report their own resource usage, but the kernel keeps it for them in the cgroup files you saw in Namespaces, cgroups and Container Internals. cAdvisor (container Advisor) reads those files for every container on the host and publishes them in Prometheus format on /metrics. Prometheus is a time-series database that scrapes (pulls) that page at a fixed interval and stores the numbers with labels. Grafana queries Prometheus and draws panels.

Quick look first. The built-in command reads the same cgroup data once:

CONSOLE
$ docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'
NAME       CPU %     MEM USAGE / LIMIT
redis      0.48%     5.465MiB / 11.63GiB
postgres   0.00%     32.49MiB / 11.63GiB

Useful, but it is a snapshot, and it only exists while you are looking. The stack below remembers.

A Compose file for the stack

YAML
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.52.1
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:v3.5.0
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=15d
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:12.2.0
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - grafana-data:/var/lib/grafana
    ports:
      - "127.0.0.1:3000:3000"
    depends_on:
      - prometheus
    restart: unless-stopped

volumes:
  prom-data:
  grafana-data:

A few details worth understanding rather than copying:

  • cAdvisor mounts parts of the host read-only because it needs to see the host's cgroups and Docker's data directory. Treat it as a privileged component and do not publish its port to the internet; Prometheus reaches it over the Compose network as cadvisor:8080.
  • The Prometheus and Grafana ports are bound to 127.0.0.1, so they are reachable only from the host itself (use an authenticated reverse proxy for anything wider; see DNS, Proxies and Firewalls).
  • Retention (--storage.tsdb.retention.time=15d) decides how much disk Prometheus uses. Without a limit it keeps 15 days anyway, but state it so it is a decision. The data lives in the prom-data volume, so it survives container replacement.

The scrape configuration, prometheus.yml:

YAML
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: cadvisor
    static_configs:
      - targets: ["cadvisor:8080"]
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

And a provisioned data source for Grafana, so it works at first start without clicking through the UI. Save it as grafana/provisioning/datasources/prometheus.yml:

YAML
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Provisioning files are the "infrastructure as code" way of doing Grafana: data sources, dashboards and alert rules can all live in Git next to your Compose file.

The queries worth knowing

Prometheus has its own query language, PromQL. Counters (numbers that only go up, like total CPU seconds) are read with rate(), which turns them into "per second" values. These queries cover the questions you ask most often. cAdvisor labels each series with the container name, so name!="" filters out the host-level entries.

Question PromQL
CPU cores used by each container sum by (name) (rate(container_cpu_usage_seconds_total{name!=""}[1m]))
Memory in use (the number the OOM killer cares about) container_memory_working_set_bytes{name!=""}
Memory as a fraction of the limit container_memory_working_set_bytes{name!=""} / (container_spec_memory_limit_bytes{name!=""} > 0)
Network received per second sum by (name) (rate(container_network_receive_bytes_total{name!=""}[1m]))
CPU throttling (share of periods throttled) rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m])
OOM kills increase(container_oom_events_total{name!=""}[1h])

Three of them deserve a note. Working set memory is "memory in use that cannot be reclaimed easily", which is what counts against the limit; plain memory_usage_bytes includes disk cache and looks scarier than it is. The limit fraction query uses > 0 to drop containers without a limit (cAdvisor reports 0 for them). And the throttling ratio is the signal people miss: a container with --cpus 0.5 can be slow while its CPU graph looks calm, because it is being paused at the cap. A ratio near 1 means the limit is too low (see Resource Limits).

One caveat: on Docker Desktop (Windows and macOS) cAdvisor runs inside the hidden VM and often cannot attach container names. The stack works best on a Linux host, which is also where production runs.

Alerts from metrics

An alerting rule is a PromQL expression plus a duration. A simple one for the OOM problem:

YAML
groups:
  - name: containers
    rules:
      - alert: ContainerOomKilled
        expr: increase(container_oom_events_total{name!=""}[10m]) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.name }} was OOM-killed"

Prometheus sends firing alerts to Alertmanager, which groups them, silences duplicates and routes them to email, chat or a pager. Alert on symptoms users feel (error rate, latency, a service down) and use resource alerts like this one as supporting detail. Alerts for "CPU above 80 percent" mostly create noise.

Metrics from your own application and from Docker

cAdvisor sees resources, not business behavior. Your application should expose its own /metrics page with request counts, latency histograms and queue sizes. Client libraries exist for every language; the page is plain text in this format:

TEXT
# HELP http_requests_total Total HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1027
http_requests_total{method="POST",status="500"} 3

Add a job to prometheus.yml that points to app:8000, and the numbers join the same dashboards. The Docker daemon can publish its own metrics too (container counts by state, image pulls, build durations) if you set {"metrics-addr": "127.0.0.1:9323"} in daemon.json and restart the daemon; they are named engine_daemon_*.

Mind label cardinality. Every distinct combination of label values is a separate time series in memory and on disk. Labels like method or status have a handful of values and are fine. A label holding a user ID, a URL with parameters or a request ID creates millions of series and can bring Prometheus down. Put such details in logs and traces, never in metric labels.

Traces with OpenTelemetry

A request to an online shop may pass through a gateway, an API, a cache and a database. If it is slow, a metric says "latency is up" but not which hop is to blame. A trace records the whole journey: it is a tree of spans, one per operation, each with a start time, a duration and attributes. All spans of one request share a trace ID, which is passed from service to service in an HTTP header (traceparent).

OpenTelemetry (OTel) is the vendor-neutral standard for producing this data. The pieces:

  • An SDK or auto-instrumentation in your application creates spans for incoming requests, database calls and outgoing HTTP calls.
  • The OTLP protocol sends them over gRPC (port 4317) or HTTP (port 4318).
  • A Collector receives, processes (batches, filters, samples) and forwards the data to one or more back ends, so your applications know only one address.
  • A back end such as Jaeger or Grafana Tempo stores and displays traces.

Most languages let you switch it on with environment variables alone, which fits containers well. In your Compose file:

YAML
services:
  app:
    image: myorg/shop-api:1.4.0
    environment:
      OTEL_SERVICE_NAME: shop-api
      OTEL_EXPORTER_OTLP_ENDPOINT: http://collector:4317
      OTEL_EXPORTER_OTLP_PROTOCOL: grpc
      OTEL_TRACES_SAMPLER: parentbased_traceidratio
      OTEL_TRACES_SAMPLER_ARG: "0.1"
    depends_on:
      - collector

  collector:
    image: otel/opentelemetry-collector-contrib:0.137.0
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector.yaml:/etc/otelcol/config.yaml:ro
    depends_on:
      - jaeger

  jaeger:
    image: jaegertracing/jaeger:2.10.0
    ports:
      - "127.0.0.1:16686:16686"

The sampler lines matter: tracing every request is expensive at scale, so parentbased_traceidratio with 0.1 keeps about 10 percent of traces, and follows the decision of the calling service so a trace is never cut in half. The collector configuration connects a receiver, a processor and an exporter into a pipeline:

YAML
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]

insecure: true is acceptable here only because the traffic stays on the private Compose network. Open http://localhost:16686, pick the shop-api service and you see each trace as a waterfall of spans; a slow database call shows up as one long bar. OTel can also carry metrics and logs through the same collector, so many teams move to it for all three signals over time.

A sensible order of work

  1. Make containers log to stdout and keep logs rotated (Logging and Monitoring).
  2. Add cAdvisor, Prometheus and Grafana, and set resource limits so the graphs have a ceiling to compare against.
  3. Add /metrics to your own services, then alerts on user-visible symptoms.
  4. Add tracing once you have more than two services calling each other.
  5. Keep the dashboards and alert rules in Git, and test restores of the Prometheus volume like any backup (Backup and Restore).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A container limited to --cpus 0.5 feels slow, but its CPU usage graph looks low. Which metric most directly reveals the problem?

    When a container hits its CPU quota the kernel pauses it until the next period. A throttling ratio near 1 shows it is being held back, even if average CPU looks modest.
  2. 2. Which memory metric is the best match for what the OOM killer compares against the limit?

    The working set is memory in use that cannot be reclaimed easily. The plain usage metric includes reclaimable cache and overstates the pressure.
  3. 3. Why should a user ID or full URL never be used as a Prometheus label value?

    High-cardinality details belong in logs and traces. Metric labels should have a small, bounded set of values.
  4. 4. What is the job of an OpenTelemetry Collector in the tracing setup?

    Applications send OTLP to the collector, which decouples them from the back end. Jaeger or Tempo stores and displays the traces.
  5. 5. How does Prometheus normally get data from cAdvisor?

    Prometheus is pull-based: scrape_configs list targets, and each one is read on every scrape interval.

Summary

  • Logs say what happened, metrics say how much over time, traces say where a request spent its time; use all three.
  • cAdvisor turns cgroup data into per-container Prometheus metrics, Prometheus scrapes and stores them, and Grafana draws them (provision data sources and dashboards as files).
  • Start with queries for CPU, working-set memory, memory vs limit, throttling and OOM events; alert on symptoms users feel.
  • Expose your own /metrics, keep label cardinality low, and consider Docker's metrics-addr for daemon metrics.
  • OpenTelemetry sends traces with OTLP to a Collector, which forwards them to Jaeger or Tempo; configure SDKs through OTEL_* environment variables and sample to control cost.
  • cAdvisor works best on a Linux host; on Docker Desktop container names are often missing.

Next: Container Storage