Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 27 ยท Projects

Project: Monitor a Fleet of Servers

Monitor a fleet of Linux servers: install node_exporter, discover hosts with file_sd, write host queries, predict full disks with predict_linear and build a dashboard.

Every application ends up on a machine, and machines fail in boring, predictable ways: the disk fills up, memory runs out, a host stops answering. In this project you monitor a small fleet of Linux servers. You install node_exporter on each one, let Prometheus find the hosts from a file you can edit without restarting anything, write the queries for CPU, memory, disk and network, and add an alert that warns you hours before a disk is full instead of after.

It combines Exporters: Node, Blackbox and Friends, Service Discovery and Relabeling, rate and counter math and alerting.

node_exporter, Prometheus and Grafana cannot run on this site, and your servers are not here either, so the commands, configuration and queries are read-only. The runnable TryIts are Python stand-ins: one turns a file_sd target file into targets with labels, and one implements the predict_linear calculation and applies the disk alert to a small simulated fleet. They model the logic; they are not Prometheus.

The plan

Four steps to host monitoring

Install node_exporter on every server.
Discover A target file lists the fleet.
Query CPU, memory, disk and network.
Predict Warn before the disk fills.

Each step is small. Together they cover most host incidents.

1 / 2

Step 1: install node_exporter

node_exporter reads the operating system's counters (from /proc and /sys) and serves them as metrics on port 9100. Install it on each server as a service under its own unprivileged user. These are read-only commands for a Linux host with systemd; replace the version with the current release from the project's download page.

Terminal
# as root, after downloading and unpacking a release for your CPU architecture
useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter
install -m 0755 node_exporter /usr/local/bin/node_exporter
INI
# /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus node exporter
After=network-online.target
Wants=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=10.0.1.11:9100 \
  --collector.textfile.directory=/var/lib/node_exporter/textfile
Restart=on-failure

[Install]
WantedBy=multi-user.target
Terminal
systemctl daemon-reload
systemctl enable --now node_exporter
curl -s http://10.0.1.11:9100/metrics | head

Two choices follow the advice from Security, Cardinality and Common Pitfalls. The exporter listens on the private address of the host, not on all interfaces, and the firewall allows port 9100 only from the Prometheus server. The textfile directory lets cron jobs and scripts add their own metrics (for example "last backup finished at") by writing .prom files, with no extra daemon.

Step 2: tell Prometheus where the hosts are

With a handful of servers you could list them in static_configs, but every new server would mean editing prometheus.yml and reloading. With file-based service discovery, Prometheus watches one or more JSON or YAML files and updates its targets when they change. A configuration management tool, a deploy script or a person can maintain that file.

YAML
# prometheus.yml
scrape_configs:
  - job_name: node
    file_sd_configs:
      - files:
          - /etc/prometheus/targets/nodes.json
        refresh_interval: 1m
JSON
[
  {
    "targets": ["10.0.1.11:9100", "10.0.1.12:9100"],
    "labels": { "env": "prod", "role": "web", "dc": "fra1" }
  },
  {
    "targets": ["10.0.2.21:9100"],
    "labels": { "env": "prod", "role": "db", "dc": "fra1" }
  }
]

Each group lists addresses and the labels they share. The labels become part of every series from those hosts, so queries and alerts can say role="db" or env="prod" without any naming tricks.

Stand-in: from target file to targets

This stand-in reads the same JSON shape and prints the targets Prometheus would build, with the automatic instance and job labels. Then it adds a server to the data to show why no restart is involved.

Python What file_sd produces

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After saving the real file, open the Targets page: new hosts appear within the refresh interval (Prometheus also picks up file changes immediately on most systems). A mistake in the JSON keeps the old targets and logs an error, so watch the logs the first time.

Step 3: the host queries

The most useful node_exporter metrics fit in a short list. Every query below is read-only, grouped by instance.

PROMQL
# CPU busy, as a fraction of all cores (0 to 1)
1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

# Memory actually available to applications, as a fraction of the total
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

# Load average compared with the number of CPU cores
node_load5 / count by (instance) (node_cpu_seconds_total{mode="idle"})

# Disk space free, per mounted filesystem (skip virtual filesystems)
node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|squashfs|overlay"}
/ node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|squashfs|overlay"}

# Network traffic in bytes per second, without the loopback device
sum by (instance) (rate(node_network_receive_bytes_total{device!="lo"}[5m]))

# Uptime in seconds
time() - node_boot_time_seconds

Some details deserve a sentence each. CPU time is a counter per mode and core, so you take the idle rate and subtract it from one. MemAvailable is better than "free" because it counts memory the kernel can reclaim from caches. Load average only means something next to the core count. The fstype filter keeps in-memory and container filesystems out of disk alerts, and node_filesystem_avail_bytes (space available to ordinary users) is the right choice over node_filesystem_free_bytes, which includes space reserved for root.

Step 4: predict a full disk

A threshold alert such as "disk above 90 percent" has a built-in problem. On a 2 TB volume, 10 percent free is 200 GB, which may last months, while on a 20 GB volume it is 2 GB, which a runaway log can eat in minutes. What you really want to know is when the disk will be full.

predict_linear(v[range], t) fits a straight line through the samples of the last range and extends it t seconds into the future. If the result is below zero, the space will have run out by then at the current pace.

PROMQL
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|squashfs|overlay"}[6h], 24 * 3600) < 0

The same trick works for any resource that fills up: queue sizes, connection pools, certificate days left (with a simple subtraction instead).

Stand-in: predict_linear on a fleet

This program implements the calculation: a least-squares line through the samples, read 24 hours after the last one. It builds six hours of samples (one every five minutes) for four simulated disks, with a little deterministic jitter, and applies the alert condition from the rule below: predicted to run out within a day and under 40 percent free.

Python Which disks will fill within a day?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the table the way an operator would. db-01 is consuming far more space per hour than web-01, but it has so much room that it is not urgent. web-01 is the one to act on: it runs out in about 21 hours. logs-01 leaks slowly and becomes a problem in a day and a half, which a longer-range query would catch. The straight line is only a guess: a disk that grows in daily steps, or that gets cleaned by log rotation, breaks the assumption, so use a window long enough to cover the pattern (6 to 24 hours) and treat the alert as a ticket, not a page.

Step 5: alert rules for the fleet

YAML
# rules/hosts.yml
groups:
  - name: hosts
    rules:
      - alert: HostDown
        expr: up{job="node"} == 0
        for: 3m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} is not answering scrapes"
          runbook_url: "https://runbooks.example.com/hosts/down"

      - alert: DiskWillFillWithin24h
        expr: |
          (
            node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|squashfs|overlay"}
            / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|squashfs|overlay"} < 0.4
          )
          and
          (
            predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|squashfs|overlay"}[6h], 24 * 3600) < 0
          )
        for: 1h
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.instance }}:{{ $labels.mountpoint }} will run out of space within a day"
          runbook_url: "https://runbooks.example.com/hosts/disk"

      - alert: DiskAlmostFull
        expr: |
          node_filesystem_avail_bytes{fstype!~"tmpfs|ramfs|squashfs|overlay"}
          / node_filesystem_size_bytes{fstype!~"tmpfs|ramfs|squashfs|overlay"} < 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }}:{{ $labels.mountpoint }} has less than 5% free"

      - alert: HostMemoryLow
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.instance }} has under 10% of memory available"

HostDown pages because a silent host also hides every other problem on it. The prediction alert is a ticket: it gives you a day to act. The last-resort DiskAlmostFull pages, since at 5 percent free the system can start failing at any moment. Memory is a ticket because the kernel reclaims cache before real trouble begins; page on out-of-memory kills, not on low free memory.

Validate before loading:

Terminal
promtool check rules rules/hosts.yml
promtool check config prometheus.yml

Step 6: the host dashboard

Create a Grafana variable instance with the query label_values(node_uname_info, instance) (multi-select, include All) and filter every panel with instance=~"$instance". A host dashboard reads best as one row of overview numbers and one row of graphs per resource.

Panel Type Query Unit
Hosts up Stat sum(up{job="node"}) none
CPU busy Time series 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[$__rate_interval])) Percent (0.0-1.0)
Memory available Time series node_memory_MemAvailable_bytes{instance=~"$instance"} / node_memory_MemTotal_bytes{instance=~"$instance"} Percent (0.0-1.0)
Disk free Bar gauge the filesystem ratio query, filtered by $instance Percent (0.0-1.0)
Disk full in Table predict_linear(...) evaluated as an instant query bytes
Network in Time series sum by (instance) (rate(node_network_receive_bytes_total{device!="lo", instance=~"$instance"}[$__rate_interval])) bytes/sec

Set thresholds on the percentage panels (yellow at 20 percent free, red at 10) so a glance tells you which host to open first. For design guidance see Designing Dashboards That Help.

Verify the project

  • The Targets page shows every host UP; up{job="node"} returns one series per host.
  • Stop node_exporter on one lab server and confirm HostDown goes pending, then firing after three minutes.
  • Fill a test volume with a large file and watch the free-space panel and the prediction react.
  • Add a host to the target file, and check it shows up without restarting Prometheus.

Extend the project

  • Add a blackbox_exporter probe per host to check that SSH or HTTP answers from outside.
  • Write backup-age metrics with the textfile collector and alert when they are stale.
  • Replace the JSON file with DNS or cloud discovery, as covered in Service Discovery and Relabeling.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main advantage of file_sd_configs over static_configs for a changing fleet?

    Prometheus watches the target files and refreshes its target list, so automation can maintain the file without touching the main configuration.
  2. 2. Why is node_memory_MemAvailable_bytes usually better than free memory for alerting?

    Linux uses spare memory for caches. Free memory looks low on a healthy machine, while MemAvailable accounts for reclaimable memory.
  3. 3. What does predict_linear(node_filesystem_avail_bytes[6h], 24 * 3600) < 0 mean?

    predict_linear fits a line to the last six hours and extends it 24 hours ahead. A negative result means the available space reaches zero before then.
  4. 4. Why is the disk prediction alert a ticket and not a page?

    A prediction is a forecast, not an outage. It gives time to react, and patterns such as log rotation can make it inaccurate, so a ticket is the proportionate response.

Summary

  • Install node_exporter as a service under its own user, bind it to a private address and allow only Prometheus to reach port 9100.
  • Use file_sd_configs so that adding a host is a one-line edit of a target file; shared labels (env, role, datacenter) make queries and alerts expressive.
  • Core host queries: idle CPU rate subtracted from one, MemAvailable over total, load per core, filesystem availability ratio and network rates.
  • predict_linear turns a trend into a deadline: alert when the disk is predicted to run out within a day and is already below a free-space ratio.
  • Page for hosts that are down or nearly full, and create tickets for forecasts and soft memory pressure.
  • Build the dashboard around an instance variable and thresholds that show the worst host first.

Next: Project: A Complete Monitoring Stack with Docker Compose.