Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Monitoring and Logs

Metrics, Dashboards and Alerting for Linux Servers

Move from looking at servers to watching them: metric types, node_exporter and Prometheus, useful alert rules, thresholds that avoid noise, uptime checks, and what to do when an alert fires.

Running top when you suspect a problem is reactive. Real operations need two more things: a history of numbers (so you can see trends and compare before and after), and alerts that wake the right person when something is genuinely wrong, before users complain. This lesson introduces the standard open-source stack (node_exporter, Prometheus, Alertmanager and Grafana), how to design alerts that people do not learn to ignore, and simple options for smaller setups.

You cannot run these services on this site. Configuration is shown read-only, and the Python stand-ins implement the underlying ideas: computing a rate from a counter, and evaluating an alert rule with a "for" duration.

What to measure

Start with the four golden signals for any service plus the basic resources:

  • Latency: how long requests take (percentiles, not averages).
  • Traffic: requests per second.
  • Errors: rate of failed requests.
  • Saturation: how full the system is.
  • Host resources: CPU, memory available, disk space and inodes, disk latency, network errors, load, time synchronisation, and systemd units in a failed state.

Metric types you will meet:

Type Example Notes
Counter total bytes received only goes up (resets at restart); use a rate
Gauge memory available now goes up and down
Histogram request duration buckets allows percentiles

The Prometheus stack in one picture

flowchart LR
    N["node_exporter<br/>:9100 on each server"] -->|scraped every 15 s| P["Prometheus<br/>stores time series"]
    P --> G["Grafana<br/>dashboards"]
    P --> A["Alertmanager<br/>routes alerts"]
    A --> M["email, chat, pager"]

Prometheus pulls ("scrapes") metrics over HTTP; nothing is pushed from the servers.

node_exporter exposes hundreds of host metrics at http://host:9100/metrics as plain text:

TEXT
node_memory_MemAvailable_bytes 1.0928e+10
node_filesystem_avail_bytes{mountpoint="/"} 2.4e+10
node_network_receive_bytes_total{device="ens3"} 8.8123e+11
node_load1 0.42

Install it from your distribution (apt install prometheus-node-exporter) or as a systemd service run by an unprivileged user, and do not expose port 9100 to the internet: allow it only from the Prometheus server in your firewall (Firewalls).

A minimal Prometheus scrape configuration:

YAML
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['web01:9100', 'web02:9100', 'db01:9100']

The query language, PromQL, turns counters into rates and aggregates them:

TEXT
rate(node_network_receive_bytes_total{device="ens3"}[5m])
100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0

The last expression predicts, from the last six hours, whether the disk will fill within a day, which is far more useful than a plain "90 percent full" alert.

Python From counter to rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Alerts that deserve to exist

Every alert should be actionable, urgent (someone must act now, not "someday"), and about a symptom where possible (users are affected) rather than every possible cause. Too many alerts teach the team to mute them. A Prometheus alerting rule:

YAML
groups:
  - name: host
    rules:
      - alert: DiskWillFillIn24h
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24*3600) < 0
        for: 30m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} will fill within 24 h"

      - alert: HostDown
        expr: up{job="node"} == 0
        for: 2m
        labels: {severity: critical}

      - alert: MemoryPressure
        expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.10
        for: 10m
        labels: {severity: warning}

      - alert: SystemdUnitFailed
        expr: node_systemd_unit_state{state="failed"} == 1
        for: 5m

The for: clause is the key anti-noise tool: the condition must hold continuously for that long before the alert fires, so a 20-second CPU spike does not page anyone. Severity decides routing in Alertmanager: critical to the pager, warnings to a chat channel or ticket queue. Add a runbook link to each alert, telling the responder what to check first; the next chapter's troubleshooting playbook is a good model.

Python Alert evaluation with a 'for' duration

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice minute 6: the value briefly rises to 12, so the pending counter resets and the clock restarts. The alert fires only when the low value has lasted three minutes in a row (minute 9). Real systems add hysteresis for the same reason.

Dashboards and simpler options

Grafana connects to Prometheus and draws graphs; import the popular "Node Exporter Full" dashboard to start, then build your own with the few panels you actually use during an incident (CPU, memory available, disk, network, errors, request latency). A good dashboard answers "is it healthy?" at a glance.

You do not always need this much:

  • Uptime checks from outside (a hosted service, or curl -fsS --max-time 10 https://example.org/health from cron on another machine) catch problems your own monitoring cannot, such as the whole network being down.
  • Netdata gives per-second dashboards and built-in alerts after a one-line install.
  • Cron plus mail: a shell script that checks disk space and mails you is still better than nothing (see cron and timers).
  • systemd's OnFailure= can start a notification unit whenever a service fails.

Whatever you use, test the alert path: stop a service on purpose and confirm that a human really receives the message.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a counter metric such as node_network_receive_bytes_total need rate() to be useful?

    The raw total is just an ever-growing number; its rate over a window is the throughput.
  2. 2. What does for: 10m in an alert rule do?

    It filters short spikes so people are paged only for sustained problems.
  3. 3. Which alert is the better design for disks?

    Prediction targets the real risk and avoids noise on large stable disks.
  4. 4. Where should node_exporter's port 9100 be reachable from?

    Metrics reveal internal details, so firewall them to the scraper.

Summary

  • Measure traffic, errors, latency and saturation, plus host resources; keep history so trends are visible.
  • Prometheus scrapes exporters such as node_exporter; PromQL turns counters into rates and predictions; Grafana draws, Alertmanager routes.
  • Good alerts are actionable and urgent, use for: to avoid noise, and link to a runbook.
  • External uptime checks, Netdata, OnFailure= and simple cron checks are lighter alternatives.
  • Test the whole alert path regularly and firewall the exporter port.

Next: Scheduling with cron and systemd Timers