Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 95 ยท Processes and the System

Disks, Mounts, Logs and Monitoring

Read mounts and disk usage, parse /proc/meminfo, analyse log files with grep and awk, follow logs with tail, understand log rotation and basic system monitoring.

When a server misbehaves, three questions come up again and again: is the disk full, is the machine out of memory, and what did the logs say? All three have the same answer shape on Linux: the system publishes the facts as plain text, either through small commands (df, free, mount) or as files you can read (/proc/meminfo, /proc/mounts, /var/log/*). And because they are text, you already own the tools to analyse them: grep, awk, sort, uniq.

This lesson shows how disks and mounts fit together, how to read and parse the output of the monitoring commands, how logs are organised and rotated, and how to dig answers out of a log file. Our sandbox is a locked-down container with no /var/log and no real disks, so many examples work on sample data that we generate inside the example. The parsing techniques are exactly the same ones you use on a production server. Where a command needs a real machine, the block is labelled read-only.

Disks, partitions and mount points

Linux has no drive letters. A disk is split into partitions, each partition holds a filesystem (ext4, xfs, btrfs...), and a filesystem becomes visible by being mounted at a directory, its mount point. Everything below that directory then lives on that filesystem.

TEXT
   /dev/sda  (the disk)
    โ”œโ”€โ”€ /dev/sda1  ext4  โ”€โ”€mounted atโ”€โ”€โ–บ  /
    โ”œโ”€โ”€ /dev/sda2  ext4  โ”€โ”€mounted atโ”€โ”€โ–บ  /home
    โ””โ”€โ”€ /dev/sda3  swap
   /dev/sdb1  xfs   โ”€โ”€mounted atโ”€โ”€โ–บ  /var/lib/data
   tmpfs (RAM)      โ”€โ”€mounted atโ”€โ”€โ–บ  /tmp

Two things follow from this. A path like /home/ada/notes.txt can live on a different physical device than /etc/hosts, and a "disk full" error applies to one filesystem, not to the whole tree: /var can be full while /home has plenty of room.

The commands to look at all this:

CONSOLE
$ lsblk                  # block devices and partitions as a tree
$ findmnt                # mounted filesystems as a tree
$ mount | column -t      # every mount (long, noisy)
$ sudo fdisk -l          # partition tables (needs root)

Read-only: lsblk needs /sys/dev/block, which the sandbox does not have, and fdisk needs root. The kernel's own list of mounts, however, is always readable: /proc/mounts. Each line has six fields: device, mount point, filesystem type, options, and two obsolete numbers (dump and fsck order).

Bash Parsing a mount table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice how many of the entries are not disks at all: proc, tmpfs and others are virtual filesystems that live in memory or expose kernel data. When you are hunting for disk problems, filter them out, as the third example does.

How full is it? df

df (disk free) reports, per filesystem, size, used and available space. The most useful options are -h (human-readable sizes), -i (inodes instead of blocks), -T (show the type) and -P (POSIX format: one line per filesystem, never wrapped, which is the only format you should parse in a script).

CONSOLE
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   41G  6.2G  87% /
/dev/sda2       200G  120G   71G  63% /home
tmpfs           2.0G  1.1M  2.0G   1% /tmp

(That table is an illustration, not output from the sandbox.) Long device names make plain df wrap lines in some versions, which breaks awk; df -P avoids it. Let's build a small disk watchdog that works on that format.

Bash A disk-space watchdog on df -P output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The watchdog compares numbers, so the use + 0 forces awk to treat "88" as a number. Real scripts run something like this from cron and send mail (or ring a pager) when a line is printed.

Two ways a filesystem can be "full" are worth knowing:

  • Out of blocks: the usual case. df -h shows 100%. The culprit is usually a runaway log or a forgotten core dump; find it with du (see Disk Usage and File Information).
  • Out of inodes: every file needs an inode, and the number of them is fixed when the filesystem is created. Millions of tiny files (a mail queue, session files) can exhaust them while df -h still shows free space. df -i shows the inode columns (Inodes, IUsed, IFree, IUse%). Writes then fail with "No space left on device" even though there is space.

Deleting a file that a running process still has open does not free its space until the process closes it or exits. If df stays full after you removed a huge log, restart the process that was writing it (or truncate the file instead of deleting it with : > file).

Memory and load: free, uptime, vmstat and /proc

Three small commands cover most "is the box healthy" questions:

Command Answers Notes
free -h How much RAM and swap is used? Look at the available column, not "free": Linux uses spare RAM for caches, which it gives back on demand.
uptime How long has it run, and what is the load? Three load averages: last 1, 5 and 15 minutes. Compare to the number of CPUs (nproc).
vmstat 1 5 What is the system doing right now? Five lines, one per second: processes waiting (r), swapping (si/so), I/O (bi/bo), CPU split (us sy id wa).

Their numbers change every second, so we only print the headers in the sandbox. Their sources are plain text files you can read directly:

Bash Memory and load straight from /proc

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The same pattern, "pick the key, take the second field", reads almost anything under /proc (see The Linux Filesystem and Processes). Parsing /proc is more robust than parsing free, because the column layout of free has changed between versions.

Other tools you will meet (all read-only here, since they are not installed or need real hardware): top and htop (live process viewer), iostat (disk throughput), sar (history of CPU, memory and I/O, from the sysstat package), ss (sockets, see Networking Basics) and lsof (which process has which file open).

Where logs live

Historically each program wrote a text file under /var/log; modern systems also have the systemd journal, a binary database read with journalctl. A typical server has both:

Path Contents
/var/log/syslog or /var/log/messages General system messages (Debian/Ubuntu or Red Hat family)
/var/log/auth.log or /var/log/secure Logins, sudo, SSH attempts
/var/log/kern.log Kernel messages
/var/log/nginx/, /var/log/apache2/, /var/log/mysql/ One directory per service
/var/log/wtmp, /var/log/btmp Binary login records, read with last and lastb
CONSOLE
$ journalctl -u nginx --since "1 hour ago"    # logs of one service (systemd)
$ journalctl -p err -b                        # errors since boot
$ journalctl -f                               # follow, like tail -f
$ dmesg --ctime | tail                        # kernel ring buffer
$ tail -n 50 /var/log/auth.log                # classic text log

Read-only: the sandbox has no /var/log and no systemd. dmesg runs but shows the host kernel's boot messages, which are not meaningful (or stable) for a lesson. Many distributions restrict it to root anyway.

Programs that run as services usually do not write files themselves; they print to stdout/stderr and let the system capture it, or they call syslog with the logger command from shell scripts:

CONSOLE
$ logger -t backup -p user.notice "nightly backup finished"
$ journalctl -t backup

-t sets the tag (program name) and -p the facility and severity. Syslog severities, from worst to least: emerg, alert, crit, err, warning, notice, info, debug. In the sandbox logger has no syslog daemon to talk to:

Bash logger without a syslog daemon (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The run prints logger: socket /dev/log: No such file or directory and exits with status 1. On a real machine the call succeeds. A script that must also work where there is no syslog should write its own timestamped lines to a file; we do exactly that in the next section.

Analysing a log file

Whatever the format, log analysis follows the same recipe: filter (grep), extract the field you care about (awk, cut), then group and count (sort | uniq -c | sort -rn). First, a sample application log with one message per line and the fields date time level service message:

Bash Counting levels, services and busy hours

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Reading the result: of 36 lines, 21 are INFO, 10 ERROR and 5 WARN; web has 6 of the errors and db 4; and the two error messages are equally common (5 each), so their order in the output is just sort's last-resort tie-break (do not rely on it).

The pipeline sort | uniq -c | sort -rn is the single most useful idiom for logs: uniq only collapses adjacent equal lines, so you must sort first; -c prefixes the count; the second sort puts the biggest group on top. See sort, uniq, cut, tr and wc and Building Text Pipelines.

For time windows, ISO-8601 timestamps (2026-03-14 09:15:00) are a gift: they sort alphabetically in chronological order, so a plain string comparison selects a range:

Terminal
awk '$1" "$2 >= "2026-03-14 09:00:00" && $1" "$2 < "2026-03-14 10:00:00"' app.log

When the log has a different date format (Mar 14 09:15:01, Apache's 14/Mar/2026:09:15:01), converting it to something sortable first is half of the job.

Following a log while it grows

tail -f file (or tail -F, which also survives rotation, below) prints new lines as they are appended, and is how you watch a service live. Press Ctrl-C to stop. In a script you usually bound it with timeout:

Bash tail -f with a background writer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Status 124 is how timeout says "I had to kill it". Useful variants: tail -n 100 -f starts with the last 100 lines; tail -f a.log b.log follows several files with headers; grep --line-buffered lets you filter a live stream (tail -f app.log | grep --line-buffered ERROR) without the buffering delay described in Pipes and Pipelines.

Log rotation

A log file that only grows will eventually fill the disk. Rotation renames the current file (app.log to app.log.1), starts a fresh one, optionally compresses old copies, and deletes the oldest. On most distributions the logrotate program does this from a daily cron job or systemd timer, driven by small config files:

INI
# /etc/logrotate.d/myapp   (read-only example)
/var/log/myapp/*.log {
    weekly
    rotate 4
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

That reads: rotate weekly, keep four old logs, compress them (but leave the newest old one uncompressed for a cycle, delaycompress), do not fail if the file is missing or empty, and use copytruncate for programs that keep the file open: instead of renaming the file (the program would keep writing into the renamed one), it copies the content and empties the original in place. The cost: a few lines written during the copy can be lost.

The logic is simple enough to write yourself. Here is a rotation function with a keep-count:

Bash Rotating a log by hand, keeping three copies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

After five rotations only three archives remain: the two oldest days were deleted. A real logrotate adds locking, permissions, a postrotate hook to tell the service to reopen its log (usually systemctl reload or kill -HUP), and size limits. The lesson for your own services: always bound your logs, either with logrotate, or by logging to the journal or a pipe that rotates for you.

Never rm an active log file and expect the disk space back (see the tip above), and never rotate by mv alone while a process still holds the file open: it keeps writing into the renamed file. Either signal the process to reopen its log, or use copytruncate.

The monitoring toolkit: lscpu, mpstat, iostat, iotop, atop, nmon and iftop

Besides free, uptime and vmstat, administrators reach for a handful of specialised monitors. Most come from the sysstat package or are separate small programs. None of them is installed in the sandbox, so the table is read-only, but the next example rebuilds what they compute from /proc.

Tool Shows Typical use
lscpu CPU model, sockets, cores, threads, caches, flags lscpu; one-time hardware facts
mpstat -P ALL 2 5 CPU use per core: %usr, %sys, %iowait, %idle Find one saturated core hidden by a low average
iostat -x 2 Per-disk throughput, queue length, await (ms per request), %util A high %util with a high await means the disk is the bottleneck
iotop -o Which process is reading or writing the disk (needs root) Find the program that hammers the disk
atop Processes, CPU, memory, disk and network in one screen, with history logs Look back at what happened at 03:00
nmon Interactive dashboard: press c, m, d, n for CPU, memory, disks, network Quick overview, also writes CSV
iftop / nload Network use per connection or interface (needs root) Who is eating the bandwidth
free -h, vmstat -s Memory summary, counters since boot Quick checks

Two habits make these tools readable. First, the first report is an average since boot: iostat and mpstat print one summary line immediately and only the following lines show the current interval, so always give an interval (iostat -x 2 3). Second, always ask which resource is saturated before you look at processes: CPU (%usr + %sys high), disk (%iowait and await high), memory (si/so in vmstat, little MemAvailable) or network (an interface near its link speed).

All of them read the same kernel counters you can read yourself. Counters only grow, so a rate is always the difference between two samples divided by the time between them:

Bash Compute CPU, disk and network rates from /proc samples

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

lscpu itself reads /sys and /proc/cpuinfo; the sandbox has no /sys, so it fails here, but the logical CPU count comes straight from /proc/cpuinfo. In the /proc/stat line the columns are user nice system idle iowait irq softirq steal, in clock ticks. The fixed samples show a 600-tick interval in which the CPU was 50% idle and 8.3% waiting for I/O, which is exactly what mpstat would print.

Reading the columns

The reports use compact column names. Here are the ones people ask about most.

iostat prints a CPU block and a device block. iostat -c shows only CPU, -d only devices, -x the extended device columns, -k or -m the units (kilobytes or megabytes), -h human-readable, and iostat -x 2 5 takes five samples two seconds apart.

Column Meaning
%user, %nice, %system, %iowait, %steal, %idle Share of CPU time in user code, niced user code, the kernel, waiting for I/O, stolen by the hypervisor (virtual machines) and idle
tps Transfers (requests) per second
kB_read/s, kB_wrtn/s, kB_dscd/s Kilobytes per second read, written and discarded (TRIM)
kB_read, kB_wrtn, kB_dscd Totals since boot (first report) or in the interval
r/s, w/s, rrqm/s, wrqm/s Reads and writes per second; reads and writes merged per second
r_await, w_await, aqu-sz Average time per request in ms, including queueing; average queue length (older versions: await, avgqu-sz, svctm, avgrq-sz)
%util Share of time the device was busy; near 100% means saturated (for SSDs it is only a rough guide)

lscpu summarises the processor: lscpu -e lists one row per logical CPU (core, socket, online state, max MHz), lscpu -p prints the same as parseable CSV, lscpu --json emits JSON for scripts, lscpu --caches shows cache sizes, lscpu -x prints masks in hex and lscpu -b or -c select only online or offline CPUs. vmstat -m (or /proc/slabinfo) lists kernel slab caches, the pools the kernel uses for small objects; look there when "kernel memory" grows.

ps and top show %CPU and %MEM (shares of one CPU and of physical RAM), VSZ (virtual size) and RSS (resident size in RAM), STAT (state), TIME (CPU time used) and in top, TIME+ (the same with hundredths of a second). In top, press z for colour, 1 for a line per CPU, M or P to sort by memory or CPU, k to kill and q to quit. htop adds mouse use, a tree view (F5), filtering and an easy kill menu. glances is a Python tool that shows CPU, memory, disk, network, sensors and the top processes on one screen, can export to files or a web page (glances -w) and flags values that cross thresholds in colour; install it with pip or your package manager. pmap and atop are covered in Processes and above.

du and df answer different questions: df asks the file system how full it is, du adds up the files you can see. Useful du options: -s one total per argument, -h human sizes, -c a grand total, -S do not include subdirectory sizes in a directory's own size, -B 1M use 1 MiB blocks, -0 end lines with a NUL byte for xargs -0, --max-depth=1 or -d 1 one level. Useful df options: -h human sizes, -i inodes instead of blocks, -T file system type, -l local file systems only, --total add a grand total, --sync sync before measuring.

Bash du and df options on a small tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

A small monitoring checklist

Real monitoring uses systems like Prometheus, Grafana, Nagios or a cloud provider's agent. But a handful of shell checks catches most disasters, and understanding them makes the big tools less magical. Run these (or their equivalents) regularly:

  1. Disk: df -P per filesystem and df -i for inodes, alert at 80-90%.
  2. Memory: MemAvailable in /proc/meminfo, plus swap activity from vmstat (si/so above zero for a long time means trouble).
  3. Load: uptime compared with nproc; a 1-minute load persistently above the CPU count means queued work.
  4. Processes: is the service up (pgrep, systemctl is-active, a health-check URL)? See Processes.
  5. Logs: the number of ERROR lines in the last N minutes; new entries in auth.log for failed logins.
  6. Time: a drifting clock breaks TLS, Kerberos, logs and cron. timedatectl shows the sync state.

Wrap each check in a function that prints one line and returns a non-zero status on failure, then call them from cron or a systemd timer (see Packages, Services and systemd). The exit codes from Exit Codes, set -e and Error Handling are what monitoring systems consume.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is df -P the right choice when a script parses disk usage?

    -P selects the portable POSIX format, in which long device names never wrap onto a second line. Human-readable output is -h, and inodes are -i.
  2. 2. df -h shows 40% used on /var, but programs fail with No space left on device. What is the most likely explanation?

    Every file needs an inode and the number is fixed at creation. Millions of small files can use them all while plenty of blocks remain. df -i shows the inode usage.
  3. 3. Which pipeline shows the most frequent messages first?

    uniq only collapses adjacent duplicates, so the input must be sorted first. -c adds the counts, and the final sort -rn puts the largest count on top.
  4. 4. You deleted a 20 GB log with rm, but df still shows the disk full. Why?

    The data is freed when the last name and the last open file descriptor are gone. Restart the process that writes the log, or truncate with : > file next time.
  5. 5. What does copytruncate in a logrotate configuration do?

    Renaming would leave the program writing into the renamed file. copytruncate keeps the original file (and inode) and empties it after copying, at the risk of losing a few lines written in between.

Summary

  • Filesystems are mounted at directories; "disk full" is per filesystem. /proc/mounts lists them (six fields per line), many of them virtual.
  • Use df -P for scripts, df -i for inodes, and keep in mind that a deleted file whose process is still running keeps using space.
  • Memory and load come from /proc/meminfo, /proc/loadavg, free, uptime and vmstat; look at available memory, and compare load to nproc.
  • Logs live in /var/log and in the systemd journal (journalctl); logger writes to syslog from scripts.
  • Log analysis is grep to filter, awk/cut to extract, and sort | uniq -c | sort -rn to count; ISO timestamps sort as plain strings.
  • tail -f follows a growing file; rotation (renaming, compressing and deleting old copies, with logrotate) keeps logs from filling the disk.
  • Real monitoring is built from small checks like these: disk, inodes, memory, load, services and log errors.

Next: SSH, scp and Working Remotely