Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 38 ยท Real-world Tasks

System Administration Text Tasks with sed and AWK

Everyday sysadmin jobs with sed and AWK: parse /etc/passwd, df and ps output, find failed SSH logins, edit hosts and cron files, and build reports from command output.

System administration is largely text processing. Users live in /etc/passwd, disks are reported by df, processes by ps, and every login attempt lands in a log. The tools print columns; you decide which columns matter. sed and AWK turn "scroll through 400 lines" into "one number" or "three lines".

The sandbox cannot show your machine's real files, so the examples below use sample data that has the same format as the real thing. The commands you learn work unchanged on a real server; where a command only makes sense on a real system, it is shown read-only and marked as such.

Parsing /etc/passwd (colon-separated)

Each line of /etc/passwd has seven colon-separated fields: name:password:UID:GID:comment:home:shell.

Bash Users, shells and system accounts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The same works on /etc/group, /etc/shadow (root only) and any colon-separated file. A few real-system examples you can try on a Linux machine (read-only here):

Terminal
awk -F: '$3 == 0 { print $1 }' /etc/passwd          # accounts with UID 0
awk -F: '($2 == "" ) { print $1 }' /etc/shadow       # accounts with an EMPTY password (needs root)

Disk usage from df

df -P (POSIX format) prints one line per filesystem, columns: Filesystem, 1024-blocks, Used, Available, Capacity, Mounted on. The Use% column has a percent sign, which AWK needs help with.

Bash Filesystems that are filling up

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

"79%" + 0 is 79: when AWK converts a string to a number it reads the leading digits and stops at the %. That is why $5 + 0 > 75 works without removing the sign.

Processes from ps

Bash Memory-hungry processes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

On a real machine, pipe ps aux into the same programs. Remember that ps output includes the header line (skip it with NR > 1) and that the command column can contain spaces, so it spans several fields.

Failed SSH logins in the auth log

Security logs are plain text. Counting failures by source address reveals brute-force attacks.

Bash Who is guessing passwords?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first command finds the word from and takes the following field. That is more robust than a fixed field number, because "invalid user" lines have two extra fields. On a real server the log is /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL), or journalctl -u ssh on systemd systems. Banning IPs automatically is a job for tools such as fail2ban, not for a shell one-liner.

Editing system files safely

Always work on a copy, check the result with diff, then replace the original. Many files have a tool that validates them (visudo for sudoers, nginx -t, sshd -t).

Bash Edit a hosts file copy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Cron-like and crontab text

Crontab lines have five time fields followed by a command. AWK can list what runs and when.

Bash Reading a crontab

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Working safely

  • Test every command on sample data or a copy before running it on live files.
  • Read-only commands (awk and sed without -i, with output to another file) cannot damage anything.
  • When a command edits a system file, keep a backup: cp -p file file.bak (or sed -i.bak).
  • Do not run commands built from log content or user names without quoting them; attackers can put shell characters into usernames.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does awk '$5 + 0 > 75' work on values like 79%?

    String-to-number conversion uses the leading numeric part, so "79%" + 0 is 79.
  2. 2. Which field separator splits /etc/passwd lines into their fields?

    passwd entries are colon-separated, so -F: gives name in $1, UID in $3 and shell in $7.
  3. 3. Why search for the word from in the SSH log instead of using a fixed field number?

    'invalid user NAME' adds two words, so the IP is not always in the same field.
  4. 4. What is the safest way to edit a system configuration file with sed?

    A copy plus diff plus validation lets you check the change before it can break anything.

Summary

  • /etc/passwd, /etc/group and similar files are colon-separated: use -F:.
  • df, ps and other command output has a header line (skip with NR > 1) and columns you can sum or filter; "79%" + 0 is 79.
  • Search for a keyword field (from) when line layouts vary.
  • Edit system files via copy, diff and validation, with backups.
  • Use sample data to develop a command, then run it on the real system.

Next: Bulk Edits and Refactoring Source Code