Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 27 ยท Projects

Project: A Complete Monitoring Stack with Docker Compose

Run Prometheus, Grafana, Alertmanager, node_exporter and cAdvisor with Docker Compose, provision the data source and dashboards as files, and validate it all.

The best way to learn a monitoring stack is to run a whole one. In this project you describe five services in a single Compose file: Prometheus to collect and store metrics, Alertmanager to deal with alerts, Grafana to draw dashboards, node_exporter to report on the host and cAdvisor to report on the containers. Grafana's data source and dashboards come from files, so a fresh docker compose up gives you a working setup with no clicking, and you can keep everything in Git.

You will need Docker with Compose on your own machine, and the ideas from How Prometheus Works, Installing Prometheus and Your First Scrape, Grafana: Data Sources and Dashboards and Alerting Rules and Alertmanager. If Compose is new to you, the Docker tutorial covers it.

Docker cannot run on this site, so the Compose file and all configuration files are read-only; you run them on your own computer. The runnable TryIts are JavaScript stand-ins. One checks that the pieces of a stack agree with each other (names, ports, mounts and networks), and one imitates the Prometheus Targets page to show how common mistakes look. The image tags in the files are examples: look up the current release of each image and pin an exact version.

The plan

flowchart LR
  NE["node-exporter<br/>host metrics"] --> P["prometheus<br/>scrapes and stores"]
  CA["cadvisor<br/>container metrics"] --> P
  P --> AM["alertmanager<br/>routes alerts"]
  P --> G["grafana<br/>dashboards"]

Prometheus pulls from the exporters (and from itself and Alertmanager), pushes alerts to Alertmanager and answers Grafana's queries. All five containers share one network.

All services live on one user-defined network called monitoring. Inside that network Docker's built-in DNS resolves each service name to the container, so Prometheus reaches the exporters as node-exporter:9100 and cadvisor:8080. Only the web interfaces you need to open in a browser are published to the host, and only on 127.0.0.1.

Project layout

TEXT
monitoring-stack/
  compose.yaml
  prometheus/
    prometheus.yml
    rules/
      stack.yml
  alertmanager/
    alertmanager.yml
  grafana/
    provisioning/
      datasources/
        prometheus.yml
      dashboards/
        provider.yml
    dashboards/
      host-overview.json
  secrets/
    grafana_admin_password.txt

Keep secrets/ out of version control (add it to .gitignore) and put a real password in the file.

The Compose file

YAML
# compose.yaml
name: monitoring

services:
  prometheus:
    image: prom/prometheus:v3.5.0
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=15d
      - --web.enable-lifecycle
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus-data:/prometheus
    ports:
      - "127.0.0.1:9090:9090"
    networks: [monitoring]
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:v0.28.1
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager-data:/alertmanager
    ports:
      - "127.0.0.1:9093:9093"
    networks: [monitoring]
    restart: unless-stopped

  grafana:
    image: grafana/grafana:12.0.2
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
      GF_USERS_ALLOW_SIGN_UP: "false"
    secrets:
      - grafana_admin_password
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
    ports:
      - "127.0.0.1:3000:3000"
    depends_on: [prometheus]
    networks: [monitoring]
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:v1.9.1
    command:
      - --path.rootfs=/host
    pid: host
    volumes:
      - /:/host:ro,rslave
    networks: [monitoring]
    restart: unless-stopped

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.52.1
    privileged: true
    devices:
      - /dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    networks: [monitoring]
    restart: unless-stopped

networks:
  monitoring:

volumes:
  prometheus-data:
  alertmanager-data:
  grafana-data:

secrets:
  grafana_admin_password:
    file: ./secrets/grafana_admin_password.txt

Several choices deserve an explanation.

  • Ports bind to 127.0.0.1. Prometheus and Alertmanager have no sign-in by default, so do not publish them on all interfaces of a server. The exporters publish nothing at all: only Prometheus needs to reach them, through the shared network. To expose Grafana to others later, put a reverse proxy with TLS in front.
  • The Grafana password comes from a secret file. Grafana reads any setting from a file when the environment variable name ends in __FILE, which keeps the password out of the Compose file and out of docker inspect. The container's user must be able to read the file.
  • Prometheus is configured read-only. The config and rule files are mounted :ro, and the lifecycle flag lets you reload with an HTTP call. That flag also allows remote shutdown, which is only acceptable because the port is bound to localhost (see Security, Cardinality and Common Pitfalls).
  • Data lives in named volumes. Removing containers keeps your metrics, dashboards state and alert silences; docker compose down -v deletes them.
  • node_exporter and cAdvisor need host access. node_exporter shares the host process namespace and sees the root file system read-only, and cAdvisor reads container information from /var/lib/docker, /sys and /var/run, and is commonly run privileged. Only run them on machines you control. On Docker Desktop (Windows and macOS) these containers see the small Linux virtual machine that Docker uses, not your laptop's operating system, and some paths may be missing, so expect partial results there.

Prometheus, rules and Alertmanager

YAML
# prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["prometheus:9090"]

  - job_name: alertmanager
    static_configs:
      - targets: ["alertmanager:9093"]

  - job_name: node
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: cadvisor
    static_configs:
      - targets: ["cadvisor:8080"]

Every address uses a service name and the port inside the container, not a published port. From within the network, localhost would mean the Prometheus container itself, which is the most common mistake in Compose-based setups.

YAML
# prometheus/rules/stack.yml
groups:
  - name: stack
    rules:
      - alert: TargetDown
        expr: up == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.job }} target {{ $labels.instance }} is down"

      - alert: HostMemoryLow
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "Less than 10% of host memory is available"
YAML
# alertmanager/alertmanager.yml
route:
  receiver: default
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: default
    # add email_configs, slack_configs or pagerduty_configs here to be notified

The default receiver has no integration, so alerts will show up in the Alertmanager web page but nobody will be notified. Add a real receiver as in Project: Alerts with Burn Rates and Routing when you are ready.

Grafana provisioning

Provisioning files are read when Grafana starts. The first declares the data source, the second tells Grafana to load dashboard JSON files from a folder.

YAML
# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
YAML
# grafana/provisioning/dashboards/provider.yml
apiVersion: 1
providers:
  - name: stack
    type: file
    disableDeletion: true
    allowUiUpdates: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards

The data source url is again a service name. With access: proxy, the Grafana server (not your browser) sends the queries, so the address only has to work inside the Docker network. The uid: prometheus matters: dashboard JSON refers to its data source by uid, so a fixed uid makes the same dashboard file work on every machine. Put host-overview.json in grafana/dashboards/: build the dashboard in the Grafana interface (see Designing Dashboards That Help), use Share > Export to save it, and commit the file.

Validate before you start

The Prometheus and Alertmanager images contain the checking tools, so you do not need to install anything else. Run the checks in this order and fix each message before moving on.

Terminal
docker compose config --quiet
docker compose run --rm --no-deps --entrypoint promtool prometheus \
  check config /etc/prometheus/prometheus.yml
docker compose run --rm --no-deps --entrypoint amtool alertmanager \
  check-config /etc/alertmanager/alertmanager.yml

The first command renders the Compose file and fails on syntax and reference errors, the second checks the Prometheus configuration and the rule files it points to, and the third checks Alertmanager. Then start the stack and look:

Terminal
docker compose up -d
docker compose ps
docker compose logs --tail=20 prometheus

Open http://localhost:9090/targets and expect every target to be UP. Try the query up in the Prometheus UI, then open http://localhost:3000, sign in as admin with your secret password and check Connections > Data sources. After editing the Prometheus configuration, reload it without restarting:

Terminal
curl -X POST http://localhost:9090/-/reload

Stand-in: do the pieces agree?

Most problems in a stack like this are disagreements between files: a typo in a service name, a port that does not match, a rule folder that was never mounted. This program takes the facts from the files above as plain data and checks them against each other. First it checks a correct setup, then one with six typical mistakes. It models the checks; the real tools are docker compose config and promtool.

JS Check that the stack's files agree

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Each message points at one file and one line to fix. The same habit applies to real stacks: after a change, run the validation commands first, then look at the Targets page.

Stand-in: how mistakes look on the Targets page

Two mistakes from the list above are invisible to promtool, because the configuration is valid; they only show up as a target that is DOWN. This program simulates the Targets page for a stack with a misspelled service name and a wrong port, shows the errors you would read, and shows up == 0 alerting on them. Messages are shortened compared with the real ones, which also include IP addresses.

JS Reading the Targets page

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The two error texts are worth memorizing. "No such host" is a name problem: the service name is misspelled, or the two containers are not on a shared network. "Connection refused" is a port problem: the name resolved, but nothing listens on that port inside the container (remember to use the container port, not the published one).

Troubleshooting checklist

Symptom Likely cause
Target DOWN, "no such host" Wrong service name or not on the monitoring network
Target DOWN, "connection refused" Wrong port; use the port inside the container
Prometheus exits at start promtool check config finds the error; read docker compose logs prometheus
Rules never appear The rules folder is not mounted, or rule_files uses a path that does not exist inside the container
Grafana says the data source is unreachable url uses localhost instead of http://prometheus:9090
No container metrics from cAdvisor The host paths are missing (common on Docker Desktop); check docker compose logs cadvisor
Grafana login fails The secret file has a different password than you think, or it has a trailing newline problem; check the file

Extend the project

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Inside the Compose network, how should Prometheus address the cAdvisor container?

    Docker's DNS resolves service names inside a shared network. localhost would be the Prometheus container itself, and published host ports are irrelevant between containers.
  2. 2. Why are the web ports published as 127.0.0.1:9090:9090 instead of 9090:9090?

    Without the address prefix Docker publishes on all interfaces. Prometheus and Alertmanager have no authentication by default, so keep them on localhost or behind a proxy.
  3. 3. A target shows 'connection refused'. What does this tell you compared with 'no such host'?

    Connection refused means DNS worked and reached the container, but the port is wrong or the program is not listening. No such host points to a name or network problem.
  4. 4. What is the benefit of provisioning the Grafana data source and dashboards from files?

    Provisioning files are read at startup, so the stack can be rebuilt from the repository without manual configuration.
  5. 5. Which command checks the Prometheus configuration and its rule files before you start the stack?

    promtool ships in the Prometheus image, and check config also loads the rule files named in rule_files.

Summary

  • One Compose file runs Prometheus, Alertmanager, Grafana, node_exporter and cAdvisor on a shared monitoring network where service names are DNS names.
  • Use container ports and service names in scrape targets, Alertmanager addresses and the Grafana data source, never localhost or published ports.
  • Publish only the UIs you need, on 127.0.0.1, and keep passwords in secret files.
  • Provision the data source and dashboards as files with a fixed data source uid so the stack is reproducible.
  • Validate with docker compose config, promtool check config and amtool check-config before docker compose up, then confirm all targets are UP.
  • "No such host" means a name or network problem; "connection refused" means a port problem.

Next: Exercises with Solutions.