Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Monitoring and Logs

Performance Monitoring: CPU, Memory, Disk and Load

Find bottlenecks on a Linux server with top, vmstat, free, iostat and ss: read load average, memory and swap correctly, spot I/O wait, and apply the USE method step by step.

"The server is slow" is not a diagnosis. Is the CPU busy, is it out of memory, is the disk saturated, or is it waiting on the network? Linux exposes the numbers to answer that in a minute or two, if you know what to read. This lesson walks through the core tools, how to interpret their output correctly (several numbers are famously misleading), and a simple method, the USE method, that keeps you from guessing.

The site cannot show live numbers from a real kernel, so the commands are read-only and the sample output is illustrative. The runnable examples are Python stand-ins that parse typical command output and apply the interpretation rules.

The USE method

For each resource (CPU, memory, disk, network) check three things:

  • Utilisation: how busy is it (percent of time or capacity in use)?
  • Saturation: is work queuing because it cannot be served (run queue, swap, I/O queue)?
  • Errors: are there error counters (disk errors, packet drops, OOM kills)?

A resource with high utilisation and no saturation is simply working. Saturation is what users feel.

flowchart TD
    S["Server is slow"] --> L{"load average vs<br/>CPU count"}
    L -- "high, CPU idle low" --> C["CPU bound:<br/>top, pidstat"]
    L -- "high, iowait high" --> D["Disk bound:<br/>iostat -x, iotop"]
    L -- "normal" --> M{"free memory,<br/>swap activity?"}
    M -- "swapping" --> R["Memory pressure:<br/>free, vmstat, OOM log"]
    M -- "fine" --> N["Network or the app itself:<br/>ss, application logs"]

A first-pass decision tree. Always confirm with a second tool before acting.

Load average

uptime (or the top line of top) shows three numbers: the average of the number of tasks that were running or waiting to run (and, on Linux, tasks in uninterruptible sleep, usually waiting for disk) over the last 1, 5 and 15 minutes.

TEXT
 10:42:07 up 41 days,  3:12,  2 users,  load average: 6.20, 4.80, 2.10

Compare it with the number of CPU cores (nproc). On a 4-core machine, a load of 4 means every core is busy; 6.2 means about 2 tasks are waiting. The trend tells the story: 6.20, 4.80, 2.10 is getting worse; 2.10, 4.80, 6.20 is recovering. A high load with low CPU use usually means processes stuck waiting for disk or network storage.

Python Interpret the load average

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

CPU: top, htop, pidstat, mpstat

top is everywhere. Useful keys: 1 (per-CPU lines), M (sort by memory), P (by CPU), c (full command), k (kill), q (quit). Read the summary line:

TEXT
%Cpu(s): 62.0 us, 8.0 sy, 0.0 ni, 20.0 id, 9.5 wa, 0.0 hi, 0.5 si, 0.0 st
  • us user code, sy kernel code, ni niced processes.
  • id idle. wa iowait: CPU idle while waiting for disk. High wa points at storage, not CPU.
  • st steal: a virtual machine whose host is over-committed. If st is high, the problem is outside your VM.
  • hi, si: interrupts; very high si can mean network packet floods.
Terminal
htop                      # friendlier top (install it)
mpstat -P ALL 2 3         # per-core statistics (package: sysstat)
pidstat 2 5               # per-process CPU, every 2 seconds, 5 times
ps aux --sort=-%cpu | head
ps -eo pid,ppid,stat,pcpu,pmem,comm --sort=-pcpu | head

A process in state D (uninterruptible sleep) is waiting for I/O and cannot even be killed; many of them explain a high load with idle CPU. State Z is a zombie: finished, waiting for its parent to collect it.

Memory: free, swap and the page cache

Terminal
free -h
#                total   used   free  shared  buff/cache  available
# Mem:            15Gi   4.1Gi  420Mi  310Mi        11Gi       10Gi
# Swap:          2.0Gi   128Mi  1.9Gi

The most misread number on Linux is free. Linux uses spare memory as a page cache for files and releases it instantly when programs need it. So low free is normal and good. Look at available: an estimate of how much memory programs could obtain without swapping. Worry when available is small (below roughly 10 percent) and swap is being used actively.

Terminal
vmstat 2 5
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
#  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

In vmstat, r is runnable tasks (compare with cores), b blocked tasks, si/so are swap-in and swap-out per second (sustained non-zero values mean memory pressure; a little used swap that sits unchanged is harmless), bi/bo blocks in/out, and wa iowait.

When memory runs out completely the kernel's OOM killer terminates a process. Find it in the logs:

Terminal
journalctl -k | grep -i -E 'out of memory|oom'
dmesg -T | grep -i 'killed process'

Check per-process memory with ps aux --sort=-rss | head (RSS is resident memory) and smem if installed. Tune swapping with vm.swappiness (default 60; many database servers use 1 to 10) via sysctl.

Python Memory verdict from free output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Disk: df, du, iostat, iotop

Capacity first:

Terminal
df -h                      # space per filesystem
df -i                      # inodes (can run out while space remains)
du -xh --max-depth=1 /var | sort -h | tail
lsof +L1                   # deleted files still held open by processes

Throughput and latency:

Terminal
iostat -xz 2 3             # per-device extended statistics (sysstat)
iotop -oP                  # which processes do I/O (needs root)

In iostat -x focus on %util (time the device was busy; near 100 percent on a single disk means saturated, though NVMe devices handle parallel requests so it is less meaningful), r_await and w_await (average milliseconds per request: tens of milliseconds on SSDs means trouble), and aqu-sz (queue length). Slow disks plus high wa in top confirm a storage bottleneck.

Network: ss, ip -s, sar

Terminal
ss -s                              # socket summary
ss -tn state established | wc -l
ip -s link show ens3               # RX/TX errors, drops
sar -n DEV 2 3                     # throughput per interface

Increasing errors or dropped counters point at cabling, drivers or overload. Many sockets in TIME-WAIT or CLOSE-WAIT are worth investigating: lots of CLOSE-WAIT usually means an application is not closing connections.

Historical data with sysstat

A snapshot cannot answer "what happened at 03:00?". Install sysstat and enable collection (sudo systemctl enable --now sysstat). Then sar -u (CPU), sar -r (memory), sar -d (disk), sar -q (load) and sar -f /var/log/sysstat/sa12 (a specific day) read the history. For continuous dashboards see Metrics and Alerting.

Which tool, which resource

CPU top, mpstat, pidstat.
Memory free -h, vmstat, smem.
Disk df, iostat -x, iotop.
Network ss, ip -s, sar -n.

Check utilisation, saturation, errors.

1 / 2

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A 4-core server shows load average 1.2, 4.9, 6.3. What is the best reading?

    The 1-minute value is lowest, the 15-minute highest, so the load peaked earlier and is now falling.
  2. 2. free -h shows only 200 MiB free but 9 GiB available and no swapping. Is there a memory problem?

    Available, not free, estimates what programs can get without swapping.
  3. 3. High %wa in top with low user CPU most likely points to:

    iowait is idle time while waiting for I/O; confirm with iostat -x.
  4. 4. Which vmstat columns reveal active swapping?

    Sustained non-zero swap-in and swap-out show real memory pressure.
  5. 5. What does a high st (steal) value in a virtual machine mean?

    Steal time is CPU you wanted but the host did not provide.

Summary

  • Apply the USE method: utilisation, saturation and errors for CPU, memory, disk and network.
  • Compare load average with the core count and read its trend; high load with idle CPU suggests I/O waits.
  • In top read wa (I/O wait) and st (steal) separately from user and system time.
  • Use available in free, plus swap activity in vmstat, to judge memory; check the kernel log for OOM kills.
  • df -h, df -i, iostat -x and iotop diagnose disks; ss and ip -s diagnose networks; sysstat gives history.

Next: Metrics, Dashboards and Alerting