Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท Automation and Troubleshooting

Shell Scripts for Administrators: Safe Automation

Write administration scripts that fail safely: strict mode, quoting, functions, arguments, traps and cleanup, locking, logging, dry-run flags, idempotent steps and ShellCheck.

Scripts are how one administrator manages fifty servers. A script that works once on your laptop is easy; a script that runs unattended at 03:00 on production, survives odd input, leaves no mess and tells you when it fails is a craft. This lesson collects the habits that separate the two. It assumes you know basic Bash (variables, loops, conditionals); the Linux tutorial on this site covers those.

Two of the examples below are runnable Bash, executed in a sandbox with no network and no root. Anything touching real services, packages or disks stays in read-only blocks.

Start every script the same way

Terminal
#!/usr/bin/env bash
set -Eeuo pipefail
IFS=$'\n\t'
  • -e stops on the first failing command.
  • -u makes an unset variable an error (catches typos like $BACKUP_DIRR).
  • -o pipefail makes a pipeline fail if any stage fails, not just the last.
  • -E makes ERR traps work inside functions.
  • A safer IFS stops word-splitting on spaces.

set -e has quirks: it does not trigger inside if conditions, &&/|| lists or while tests, so do not treat it as magic. Explicit checks (command || { echo "failed"; exit 1; }) are still needed for the steps that matter.

Quote everything

The single most common scripting bug is an unquoted variable. rm -rf $dir/* with dir unset becomes rm -rf /*. Quote every expansion and use default or required forms:

Terminal
rm -rf -- "${dir:?dir is not set}"/cache      # abort if dir is empty or unset
name=${1:-default}                            # default value
files=( "$@" )                                # arrays keep spaces in names
for f in "${files[@]}"; do process "$f"; done

Use -- to end options before user-supplied names (a file called -rf is a classic trap), [[ ... ]] for tests, and $(command) instead of backticks. Prefer find ... -print0 | xargs -0 or find ... -exec over parsing ls.

Functions, arguments and exit codes

Structure scripts as small functions and a main. Exit with meaningful codes: 0 success, non-zero failure. Print errors to stderr.

Terminal
log()  { printf '%s %s\n' "$(date -u +%FT%TZ)" "$*" >&2; }
die()  { log "ERROR: $*"; exit 1; }
usage(){ echo "usage: $0 [-n] [-d dir] file..." >&2; exit 2; }

Parse options with getopts:

Terminal
dry_run=0; dir=/var/backups
while getopts ":nd:h" opt; do
  case $opt in
    n) dry_run=1 ;;
    d) dir=$OPTARG ;;
    h|\?) usage ;;
  esac
done
shift $((OPTIND - 1))

Dry runs, idempotence and cleanup

Idempotent means safe to run again: the second run changes nothing. Check before you act (mkdir -p, ln -sfn, grep -q ... || echo ... >> file), and offer a -n dry-run flag that only prints what would happen.

Use trap to clean up temporary files even when the script fails or is interrupted:

Terminal
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
trap 'log "failed at line $LINENO"' ERR
Bash A safe script skeleton: strict mode, trap, dry run

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Locking and logging

Make sure only one copy of a job runs, using flock on a file descriptor:

Terminal
exec 9>/run/lock/backup.lock
flock -n 9 || { echo "another run is active" >&2; exit 0; }

Log to the journal with logger -t backup "..." or by running the script from a systemd service (see timers), so output is collected without redirects. Always log start, end, duration and the number of items handled, which makes silent failures visible.

Patterns you will use constantly

Terminal
# Run a command on many hosts
while read -r host; do
  ssh -o BatchMode=yes -o ConnectTimeout=5 "$host" 'uptime' </dev/null || echo "$host FAILED"
done < hosts.txt

# Retry with a delay
retry() { local n=0; until "$@"; do (( ++n >= 5 )) && return 1; sleep $((n*2)); done; }

# Disk space guard
used=$(df --output=pcent / | tail -1 | tr -dc '0-9')
(( used < 90 )) || die "root filesystem is ${used}% full"

# Safe in-place edit with backup
sed -i.bak 's/^#\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sshd -t || { mv /etc/ssh/sshd_config.bak /etc/ssh/sshd_config; die "sshd_config invalid"; }

Note the </dev/null on ssh inside a while read loop: without it, ssh swallows the rest of the file from standard input.

Use shellcheck script.sh (a free linter) before every deployment. It catches unquoted variables, wrong tests and portability problems, and explains each warning. Format with shfmt. Test logic with the bats framework when scripts become important. Remember that scripts are code: keep them in Git, review changes, and never put passwords in them (read them from a protected file, environment or secret manager).

Bash Parse arguments with getopts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Know when to stop using Bash

Bash is excellent for gluing commands and wrong for complex data. When a script grows past about a hundred lines, needs real data structures (JSON, dates), or must run on many different systems, switch to Python or use a configuration tool such as the one in the next lesson.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does set -u protect you from?

    With -u the shell exits with an error when an unset variable is expanded.
  2. 2. Which is the safest way to delete a directory held in a variable?

    Quoting and the :? form abort when dir is empty instead of expanding to a dangerous path.
  3. 3. What is an idempotent script?

    Running it twice must be as safe as running it once.
  4. 4. Why use trap '...' EXIT?

    An EXIT trap makes cleanup of temporary files reliable.

Summary

  • Begin with set -Eeuo pipefail, but remember its exceptions and check critical steps explicitly.
  • Quote every expansion, use -- and ${var:?}, and use arrays for lists.
  • Structure scripts as functions with getopts, clear exit codes and errors on stderr.
  • Make scripts idempotent, add a dry-run mode, clean up with traps and prevent overlap with flock.
  • Lint with ShellCheck, keep scripts in version control, never hard-code secrets, and switch tools when logic outgrows Bash.

Next: Configuration Management and Idempotence with Ansible