Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Automation and Troubleshooting

A Linux Troubleshooting Playbook

Diagnose Linux problems methodically: gather facts, form a hypothesis, test one change at a time, and use checklists for boot failures, full disks, OOM kills, failed services and network outages.

When a server misbehaves, experienced administrators look calm because they follow a method, not because they remember every error message. The method is simple: observe, hypothesise, test, fix, verify, record. This lesson gives you that method and quick checklists for the incidents you will meet most: a service that will not start, a full disk, memory exhaustion, a network failure and a machine that will not boot.

Commands are read-only (they need a real system), and the Python stand-ins turn the decision trees into small programs so you can drill the reasoning.

The method

  1. Define the problem precisely. What exactly fails, for whom, since when, and what changed (deploy, update, config, traffic)? "The website is slow for users in one region since 14:10" is a problem statement; "it's broken" is not.
  2. Look before you touch. Collect evidence read-only: logs, metrics, systemctl status, ss, df. Do not restart anything until you have saved what you saw; a restart destroys evidence.
  3. Form a hypothesis that explains all the facts, and name a test that could prove it wrong.
  4. Change one thing at a time, and note it. Two simultaneous changes make the result unreadable.
  5. Verify the fix from the user's point of view, not just the server's.
  6. Write it down: a short note of cause, fix and prevention. Repeat incidents are a documentation failure.

Always ask first: what changed? Most incidents follow a change, and journalctl --since, package history (/var/log/apt/history.log, dnf history), Git history of your configuration repo and the last command answer it.

flowchart LR
    A["Symptom"] --> B["Gather evidence<br/>read-only"]
    B --> C["Hypothesis"]
    C --> D["Test one change"]
    D --> E{"Fixed?"}
    E -- no --> C
    E -- yes --> F["Verify and write up"]

A loop: each failed test narrows the search.

Checklist 1: a service will not start

Terminal
systemctl status myapp                   # state, last lines, exit code
journalctl -u myapp -b -n 100 --no-pager # why it failed
systemctl cat myapp                      # the unit as systemd sees it
systemd-analyze verify /etc/systemd/system/myapp.service
sudo -u appuser /usr/local/bin/myapp --check-config
ss -tlnp | grep 8080                     # is the port already taken?

Typical causes, in rough order of frequency: a configuration syntax error, a port already in use, a permissions or ownership problem on a file or directory (check the service user), a missing dependency or environment file, SELinux or AppArmor denying access (MAC), full disk, or a unit that fails its start-limit (systemctl reset-failed myapp). An exit status of 203 means the executable could not be run, 217 means the user is wrong, and 226 means a namespace or sandbox setting failed.

Checklist 2: disk is full

Terminal
df -h                                     # which filesystem?
df -i                                     # out of inodes instead of space?
sudo du -xh --max-depth=1 / 2>/dev/null | sort -h | tail
sudo du -xh --max-depth=1 /var | sort -h | tail
sudo lsof +L1                             # deleted files still held open
journalctl --disk-usage
sudo journalctl --vacuum-size=200M

The classic surprise: you delete a huge log but df shows no change. A process still has the file open, so the space is not freed until that process restarts or you truncate it: : > /proc/PID/fd/N, or better sudo systemctl restart service. Out of inodes (millions of tiny files, such as session or mail files) needs find plus deletion of the directory, not bigger disks. Prevent repeats with log rotation (logrotate) and a disk alert.

Python Why does df not drop after deleting a file?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Checklist 3: memory exhaustion and OOM kills

Terminal
free -h; vmstat 2 5                         # see the pressure (read the performance lesson)
journalctl -k | grep -i -E 'out of memory|oom-kill|killed process'
ps aux --sort=-rss | head
systemctl status myapp                      # "Main process exited, code=killed, status=9/KILL"

The OOM log line names the victim and shows the memory table at that moment. Decide: leak, undersized server, too many workers (for example PHP-FPM or Apache children sized larger than RAM allows), or a runaway job. Fixes: reduce worker counts, set per-service limits (MemoryMax= in the unit, so one service is killed instead of random ones), add swap as a cushion (not a cure), or add RAM. See Performance Monitoring for reading the numbers.

Checklist 4: network problems, bottom up

Work up the layers; stop at the first thing that fails.

Terminal
ip -br link; ip -br addr                  # 1. link and address
ip route                                  # 2. default gateway present?
ping -c 3 192.168.1.1                     # 3. gateway reachable?
ping -c 3 1.1.1.1                         # 4. internet by IP?
getent hosts example.org                  # 5. names resolve?
curl -sv --max-time 10 https://example.org -o /dev/null   # 6. application layer
ss -tlnp                                  # is the service listening at all?
sudo nft list ruleset                     # 7. is a firewall dropping it?
traceroute -n example.org; mtr -rn example.org            # where does it stop?
sudo tcpdump -ni ens3 port 443            # last resort: watch packets

The layered order matters: if ping 1.1.1.1 works but names fail, go straight to DNS; if the local port is not listening, the firewall is irrelevant. Compare from two vantage points (the server itself and a client) to separate "service is down" from "path is blocked".

Python Network triage helper

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Checklist 5: the machine will not boot

On a cloud VM use the provider's serial or recovery console. On hardware, use the console or rescue media.

  • Stops at GRUB or "no bootable device": bootloader or disk. Boot a rescue image, mount the root filesystem, chroot, and reinstall GRUB (grub-install, update-grub or grub2-mkconfig).
  • Drops to an emergency shell: usually a bad line in /etc/fstab for a disk that is missing. Mount with mount -o remount,rw /, fix /etc/fstab (use nofail on non-essential disks), and exit.
  • Kernel panic after an update: choose the previous kernel in the GRUB menu ("Advanced options"), then remove or reinstall the bad one.
  • Hangs on a service: boot with systemd.unit=rescue.target (add to the kernel line in GRUB), then journalctl -b -p err.
  • Forgotten root password: add rd.break (RHEL) or init=/bin/bash to the kernel line, remount the root read-write, passwd, and on SELinux systems touch /.autorelabel. This is why physical and console access must be protected.

After the incident, review the previous boot's log (journalctl -b -1) to find the original cause, not just the symptom.

Tools worth keeping handy

strace -f -p PID (system calls a process makes; reveals missing files and permission errors), lsof -p PID and lsof -i :80 (open files and sockets), ltrace, dmesg -T, journalctl, ss, tcpdump, iostat, perf top, and sosreport or sos report which bundles diagnostics for support. A habit that pays off: keep a personal runbook of commands and findings.

Troubleshooting loop

Observe Read-only evidence first.
Hypothesise One testable idea.
Change one thing Note it.
Verify As a user would.

Then write down cause, fix, prevention.

1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why gather evidence before restarting a failing service?

    Save status, logs and metrics first, then act.
  2. 2. You deleted a 20 GB log but df still shows the disk full. What is the likely cause?

    Space is released only when the last open handle closes.
  3. 3. Ping to 1.1.1.1 works but ping example.org fails. Where do you look next?

    Reaching an IP proves routing; failing names point to name resolution.
  4. 4. The system drops into an emergency shell right after boot. What is the most common cause?

    A mount that cannot succeed blocks boot unless marked nofail.

Summary

  • Troubleshoot with a loop: define, observe read-only, hypothesise, change one thing, verify, document.
  • Always ask what changed; logs, package history and Git answer it.
  • Services: systemctl status, journalctl -u, check config, ports, permissions and MAC.
  • Disks: df -h, df -i, du, lsof +L1. Memory: kernel OOM log, limits and worker counts.
  • Networks: go bottom up from link to application; boot failures use rescue modes and the previous kernel.

Next: Project: Harden a New Server