Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: Build a Server Health Report

Build a daily health report for a Linux server: collect disk, memory, load, failed units, logins and update status, score the results, and deliver it by timer with exit codes.

In this project you build something every administrator eventually writes: a small program that looks at a server, decides whether it is healthy, and tells you only what needs attention. It combines the tools of the whole track: monitoring commands, log reading, scripting, scheduling and alerting.

The real collector is a Bash script that needs a live machine, so it is shown as read-only code. The runnable parts work on sample captured output: a Bash text-processing step in the sandbox, and a Python scorer that turns the numbers into a verdict and an exit code. The design (collect, evaluate, report, schedule) transfers unchanged to the real thing.

Design

flowchart LR
    C["collect<br/>df, free, uptime,<br/>systemctl, last, apt"] --> E["evaluate<br/>thresholds"]
    E --> R["report<br/>text or mail"]
    R --> S["schedule<br/>systemd timer"]
    E --> X["exit code<br/>0 ok, 1 warn, 2 crit"]

Collection and judgement are separate, so you can test the judgement on saved sample data.

Good health reports follow rules from the monitoring lessons: report available memory rather than free, compare load with the number of CPUs, check inodes as well as space, and list failed units. Keep the output short: a long report is not read.

The real collector (read-only)

Terminal
#!/usr/bin/env bash
set -Eeuo pipefail

host=$(hostname -s)
cpus=$(nproc)
load1=$(cut -d' ' -f1 /proc/loadavg)
mem_total=$(awk '/MemTotal/ {print $2}' /proc/meminfo)
mem_avail=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
root_used=$(df --output=pcent / | tail -1 | tr -dc '0-9')
root_inodes=$(df --output=ipcent / | tail -1 | tr -dc '0-9')
failed_units=$(systemctl --failed --no-legend --plain | awk '{print $1}')
reboot_needed=$([ -f /var/run/reboot-required ] && echo yes || echo no)
updates=$(apt list --upgradable 2>/dev/null | grep -c upgradable || true)
failed_logins=$(journalctl -u ssh --since "24 hours ago" --no-pager 2>/dev/null | grep -c "Failed password" || true)

printf 'host=%s cpus=%s load1=%s mem_total_kb=%s mem_avail_kb=%s root_used=%s root_inodes=%s\n' \
  "$host" "$cpus" "$load1" "$mem_total" "$mem_avail" "$root_used" "$root_inodes"
printf 'failed_units=%s\nreboot_needed=%s updates=%s failed_logins=%s\n' \
  "${failed_units:-none}" "$reboot_needed" "$updates" "$failed_logins"

Everything comes from files and commands covered earlier: /proc/loadavg and /proc/meminfo are what uptime and free read, df --output prints one clean field, and systemctl --failed lists broken units. The || true after grep -c stops set -e from aborting when the count is zero (grep exits 1 when it finds nothing).

Step 1: parse the output with text tools

This block runs in the sandbox on a sample df listing. It finds filesystems above a threshold, a task you will do constantly.

Bash Find full filesystems with awk

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Step 2: evaluate with thresholds

Thresholds turn numbers into three states: OK, WARNING and CRITICAL. Put them in one dictionary so they are easy to tune, and make every check return a message so the report explains itself.

Python The health evaluator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The exit code is the contract with the rest of your automation: 0 healthy, 1 warning, 2 critical. Cron, systemd and monitoring systems all understand non-zero exits.

Step 3: report and schedule

A systemd service runs the script and mails or logs the result; a timer runs it every morning (timers).

TEXT
# /etc/systemd/system/health-report.service
[Unit]
Description=Daily server health report

[Service]
Type=oneshot
ExecStart=/usr/local/bin/health-report.sh
SuccessExitStatus=0 1
TEXT
# /etc/systemd/system/health-report.timer
[Timer]
OnCalendar=*-*-* 07:30:00
RandomizedDelaySec=10min
Persistent=true

[Install]
WantedBy=timers.target

Pipe the output to mail -s "health web01" [email protected] only when the exit code is above zero (a report that always arrives is ignored; one that arrives only when something is wrong gets read), and log every run with logger -t health-report so you can see that it ran. Add an external heartbeat so you are warned if the timer itself stops.

Extensions

  1. Check certificate expiry with openssl x509 -enddate -noout and add it as another test.
  2. Record results as one JSON line per run in /var/log/health.jsonl, and graph them.
  3. Add systemctl is-active checks for the services this particular server must run.
  4. Compare with the previous run and report only what changed.
  5. Run it over SSH from a central machine for your whole fleet (see the host loop in Shell Scripts) and print one line per server.
  6. Replace the whole thing with Prometheus alert rules once you run many servers (Metrics and Alerting), and keep the script for ad-hoc checks.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the collector use MemAvailable rather than MemFree?

    Available estimates memory usable without swapping, which is the meaningful number.
  2. 2. Why is || true added after grep -c in a script using set -e?

    A count of zero is a valid result, not a failure of the script.
  3. 3. What is the benefit of exit codes 0, 1 and 2 in the health script?

    Exit status is the universal signal between programs.
  4. 4. Why mail the report only when something is wrong?

    Signal beats noise; keep an external heartbeat to detect a silent failure of the job itself.

Summary

  • A health report separates collection, evaluation, reporting and scheduling, so each part can be tested.
  • Use meaningful metrics: available memory, load per core, space and inodes, failed units, pending updates and failed logins.
  • Thresholds map numbers to OK, WARNING and CRITICAL; the worst state becomes the exit code.
  • Schedule with a timer, notify only on problems, and watch the watcher with an external heartbeat.

Next: Practice: Administration Scenarios