Skip to content
Tutorials Center
My learning

Topics

Lesson 73 of 95 ยท Projects

Project: Server Health Monitor

Build a server health monitor in Bash: read CPU, memory, load and disk usage from /proc and df, apply thresholds, alert only on changes, and run it from cron.

Servers rarely fail without warning. The disk fills up over a week, memory creeps upward for days, the load climbs when a job misbehaves. A health monitor is a small program that looks at these numbers on a schedule and tells a human when one crosses a line. Big platforms such as Prometheus, Zabbix and Nagios do this at scale, but a 100-line Bash script is often all a single server needs, and it teaches you exactly where the numbers come from.

In this project you build monitor.sh. It reads CPU, memory, load and disk usage, classifies each value as OK, WARN or CRIT, remembers the previous level so it alerts only when something changes (no repeat spam every five minutes), and runs from cron. You will use awk, arrays, exit codes, traps and cleanup and the design habits from Designing Maintainable Scripts. The scheduling part (cron) is read-only in the browser, because the sandbox has no cron daemon and no mail system. Everything else runs, using fixtures: small fake copies of /proc files, so every output on this page is identical on every run.

flowchart LR
  C["cron<br/>every 5 minutes"] --> M["monitor.sh"]
  M --> R["read /proc and df"]
  R --> T["compare with thresholds<br/>OK, WARN or CRIT"]
  T --> S["compare with the saved level"]
  S -- "level changed" --> A["send alert"]
  S -- "same level" --> Q["stay quiet, log only"]

One run of the monitor: measure, classify, compare with last time, alert only on a change.

Where the numbers come from

Linux publishes its own statistics as files under /proc. Reading them costs almost nothing and needs no extra tools:

Source What it holds We use
/proc/meminfo one Key: value kB line per memory counter MemTotal, MemAvailable
/proc/loadavg 1min 5min 15min running/total lastpid the first number: the 1-minute load average
/proc/stat first line cpu followed by counters of CPU time (user, nice, system, idle, iowait, irq, softirq, steal, ...) total time and idle time
df -P one line per mounted filesystem, in a stable POSIX format the Capacity percent and the mount point

The -P option of df matters: it forces one line per filesystem, with the same columns on every system, so long device names do not wrap onto a second line and break your parsing. Let us look at the shape of the real files first, without printing values that change on every run:

Bash Step 1: the shape of /proc and df

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Two details need care. Memory: the right number is MemAvailable, the kernel's estimate of memory that programs could use without swapping. MemFree is misleading, because Linux deliberately uses spare memory as a file cache. So used percent is 100 * (MemTotal - MemAvailable) / MemTotal. CPU: the counters in /proc/stat only ever increase, counting time since boot. A single reading says nothing about now. You need two readings and the difference: usage is the share of the time between them that the CPU was not idle. Cron gives us a natural gap between readings, so the script saves the previous reading in a state file instead of sleeping.

Step 2: measuring with fixtures

To test a monitor you need to control the numbers it sees. The trick is to make the script read from a configurable root directory (PROC_ROOT, default /proc) and a configurable df output, then point it at fake files in tests. This design pays off in every script that reads system state.

Bash Step 2: memory, load and CPU from fake /proc files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In cpu_counters, fields 2 to 9 of the cpu line are the eight time counters, and idle time is idle + iowait (fields 5 and 6), because a CPU waiting for the disk is not doing work. guest times are already included in user, so they are not added again. The line starting cpu (with a space) is the sum of all cores; cpu0, cpu1 and so on are single cores, which the pattern /^cpu / skips.

Step 3: thresholds and levels

Each metric gets two thresholds: a warning level and a critical one. A tiny function turns a number into a word. Doing the comparison in awk is deliberate: Bash arithmetic only understands integers, and values like 0.75 or 79.9 are decimals.

Why two levels? Because they mean different things to the person who reads the alert. WARN says "look at this today", CRIT says "act now". Choose thresholds from experience on your machine; a database server that normally sits at 85% memory needs different numbers from a web server. The values below, 80 and 90 percent for memory and disks, are sensible starting points.

stateDiagram-v2
  [*] --> OK
  OK --> WARN: value reaches warn
  WARN --> CRIT: value reaches crit
  CRIT --> WARN: falls below crit
  WARN --> OK: falls below warn
  OK --> CRIT: sudden jump
  CRIT --> OK: sudden recovery

Every arrow is a change of level, and every change of level sends exactly one alert.

That diagram is the whole alerting policy. The monitor stores each check's last level in a file; when the new level is different, it sends one message, and when it is the same, it says nothing. Let us test the idea on a sequence of readings before building the full script:

Bash Step 3: levels, and alerts only on a change

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Ten readings produce four alerts: up to WARN (83), up to CRIT (93), down to WARN (82) and back to OK (71). Every line marked ALERT is one message; the other six lines are silent because the level did not change. Real monitors add refinements such as a hysteresis margin (go back to OK only below 75%, not below 80%) so a value hovering at 80% does not flap between states; you can add that to level as an exercise.

Step 4: the finished script

Now put it together. Design notes before the code:

  • Testability. PROC_ROOT, STATE_DIR, NPROC, NOW, DF_FILE and the thresholds are environment variables with sensible defaults, so the same script runs in production and in tests.
  • No overlapping runs. flock -n on a lock file makes a second copy exit immediately if the first is still running, which protects you if a run hangs.
  • Atomic state. The new levels are written to a temporary file and moved into place with mv, so a crash cannot leave a half-written state file.
  • An exit status that means something. 0 all OK, 1 at least one WARN, 2 at least one CRIT. This is the convention of Nagios-style checks, and lets other tools wrap the script.
  • A pluggable alert. If ALERT_CMD names a program, it is called with the message as its only argument. Mail, a chat webhook or logger are each a three-line script.

The demo below creates the fixtures, writes the script, and then simulates four runs, five minutes apart, with the numbers changing between them:

Bash Step 4: monitor.sh in action over four runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Read the output from the top. At 10:00 the first run has no earlier CPU sample, so the CPU line says so, and nothing alerts because every level is OK. At 10:05 four checks cross a threshold (CPU, memory and load to WARN, the /data disk to CRIT), so four alerts go out, and the exit status becomes 2 because of the critical disk. At 10:10 the problems persist, so the lines are printed but no new messages are sent; sent.txt holds only the four alerts from 10:05 and, later, the four recoveries. At 10:15 every metric returns to OK and one RECOVERED message goes out per metric.

A few points about the code are worth reading twice:

  • previous is an associative array loaded from levels, one name=LEVEL line per check. Names like disk:/data contain no =, so IFS='=' read splits correctly. The mount point is taken from field 6 of df, so a mount path containing spaces would be cut short; that is rare on servers and a fair limit for a first version.
  • check is called with a new name for every disk, so adding a disk needs no code change, and a disk that did not exist before is compared against OK by the :-OK default.
  • (( worst >= 1 )) || worst=1 is written that way because under set -e a failing arithmetic test would otherwise end the script. See Exit Codes, set -e and Error Handling.
  • Nothing is printed to standard error except real problems, which keeps the cron log clean.

Step 5: checking it on a real system

Before scheduling it, prove that the metric code also works on the machine's real /proc. This shortened script contains only the memory and load calculations, and the output shows just whether each value is a number, since the real values change every time:

Bash Step 5: the metric code on the real /proc, in a cron-like environment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The env -i PATH=/usr/bin:/bin prefix reproduces what cron gives your script. This is the number one cause of "works in my terminal, fails in cron": programs that live outside the short default PATH, or variables that your login shell set up. Either set PATH at the top of the crontab or use full paths in the script.

Scheduling with cron

cron runs commands on a schedule. You edit your own table with crontab -e (the following blocks are read-only: the sandbox has no cron daemon). Each line has five time fields and then the command:

TEXT
# minute  hour  day-of-month  month  day-of-week  command
   */5     *        *           *        *        /usr/local/bin/monitor.sh >> /var/log/monitor.log 2>&1
Field Range Notes
minute 0-59 */5 means every 5 minutes
hour 0-23 9-17 is a range, 8,12,18 a list
day of month 1-31
month 1-12
day of week 0-7 0 and 7 are both Sunday

Install the script with sudo install -m 755 monitor.sh /usr/local/bin/monitor.sh. A few cron rules prevent the usual surprises:

  • Cron mails any output of a job to the owner (or to MAILTO= at the top of the crontab). That is why the line above redirects everything to a log file: otherwise, a monitor that prints a status line every run would send a mail every five minutes. Alerts go out through ALERT_CMD instead.
  • Cron's environment is minimal (see Step 5). Put PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin at the top, or use absolute paths.
  • Percent signs are special in crontab lines (they mean newline): escape them as \% inside a date +%F call, or move the command into a script.
  • Rotate the log, or it grows forever: a file /etc/logrotate.d/monitor with weekly, rotate 8, compress and missingok is enough.
  • Test the exact command line as the same user, with env -i, before trusting the schedule.

A systemd timer is the modern alternative and gives you logs in the journal (journalctl -u monitor.service). It takes two small unit files, again read-only here:

INI
# /etc/systemd/system/monitor.service
[Unit]
Description=Server health check

[Service]
Type=oneshot
ExecStart=/usr/local/bin/monitor.sh
INI
# /etc/systemd/system/monitor.timer
[Unit]
Description=Run the health check every 5 minutes

[Timer]
OnCalendar=*:0/5
Persistent=true

[Install]
WantedBy=timers.target

Enable it with sudo systemctl enable --now monitor.timer and inspect schedules with systemctl list-timers. A service that exits with status 1 or 2 is shown as failed, so the monitor's exit codes are immediately visible there. For services in general, see Packages, Services and systemd.

Alert channels

Because ALERT_CMD is just a program that receives one argument, adding a channel is easy. Two examples, both read-only because they need a mail system and a network:

Terminal
#!/bin/sh
# mail-alert.sh: send the message by email (needs a configured mail command)
printf '%s\n' "$1" | mail -s "[monitor] $(hostname): $1" [email protected]
Terminal
#!/bin/sh
# webhook-alert.sh: post to a chat webhook, and also write to the system log
logger -t monitor -p user.warning -- "$1"
curl -fsS --max-time 10 -X POST -H 'Content-Type: application/json' \
     -d "{\"text\": \"$1\"}" "$WEBHOOK_URL"

Keep the webhook URL in a root-readable file, not in the script, because anyone who has it can post in your channel. An important rule for monitoring: the monitor must not depend on the thing it watches. An alert path that goes through the same server that has just run out of disk may never leave it; for critical systems, add an outside check (a second machine or a hosted uptime service) that notices when the server stops reporting at all.

Ideas for extending it

  • More checks: a port is listening (ss -ltn), a service is active (systemctl is-active), a file is fresh (find -mmin), a backup finished last night, SSL certificate expiry.
  • Hysteresis and grace periods: only alert after the bad level persists for three runs.
  • Per-process view: list the top memory users in the alert text with ps -eo pid,comm,%mem --sort=-%mem | head.
  • History: append each reading to a CSV file and graph it later, or log it in the format of Project: Log Analyzer to reuse its tools.
  • Self-test: adopt the approach from Project: A Mini Test Framework and turn the four-run scenario into automatic assertions.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which /proc/meminfo field best answers 'how much memory can programs still use without swapping'?

    MemFree ignores memory the kernel uses as a reclaimable cache. MemAvailable is the kernel's own estimate of what is usable, so used percent is computed from MemTotal and MemAvailable.
  2. 2. Why can't you compute current CPU usage from a single reading of /proc/stat?

    The counters hold total ticks since boot. The share of non-idle time between two samples is the usage in that interval.
  3. 3. What does the monitor do when a check stays at CRIT on consecutive runs?

    Alerting on changes keeps the signal useful. The saved level file lets each run know what it reported before.
  4. 4. A script works in your terminal but fails in cron with 'command not found'. What is the most likely cause?

    Cron does not read your login profile. Set PATH in the crontab or use absolute paths, and test with env -i.
  5. 5. Why does the crontab line redirect with >> /var/log/monitor.log 2>&1?

    Cron mails whatever a job prints. Redirecting to a log keeps the mailbox empty; real alerts go through the alert command.

Summary

  • Linux exposes metrics as files: /proc/meminfo, /proc/loadavg, /proc/stat, plus df -P for disks. Use MemAvailable for memory and two /proc/stat samples for CPU.
  • Make the script testable: read from PROC_ROOT, DF_FILE and STATE_DIR variables and feed it fixtures.
  • Classify each value as OK, WARN or CRIT with two thresholds, comparing decimals in awk.
  • Save the last level of every check and alert only on a change, including a RECOVERED message.
  • Use flock -n against overlapping runs, write state atomically with a temporary file and mv, and return exit status 0, 1 or 2.
  • Run it from cron (or a systemd timer) every few minutes, redirect output to a log, and test with env -i to match cron's minimal environment.
  • The alert path must not depend on the server it watches.

Next: Linux and Bash Cheat Sheet