Skip to content
Tutorials Center
My learning

Topics

Lesson 62 of 95 ยท Advanced Bash

Designing Maintainable Scripts

Design Bash scripts that last: main function, usage text, logging, layered configuration, dry-run mode, idempotency, locking, tests and when to switch language.

A ten-line script that you wrote for yourself is easy to understand. A two-hundred-line script that three people edit, that runs from cron at night and that deletes things is a different animal. It needs a shape: a place for every kind of code, a predictable way to be configured, a safe way to try it out, and a way to prove that it still works after the next edit.

This lesson collects the design habits that professionals apply to shell scripts. None of them is clever; together they are the difference between a script people trust and one they are afraid to touch. It builds on Exit Codes, set -e and Error Handling, Functions and Script Arguments and Options.

The shape of a script

A well-structured script reads top to bottom like a short document:

TEXT
 #!/usr/bin/env bash        <- 1. shebang
 set -euo pipefail          <- 2. strict mode
 readonly ...               <- 3. constants and defaults
 usage()  log()  die()      <- 4. small helpers
 do_something() ...         <- 5. one function per job
 main()                     <- 6. wires everything together
 main "$@"                  <- 7. the only line that "does" anything

The most important idea is the last line. Everything above it only defines things; nothing runs until main "$@". That gives you three benefits: functions can be written in any order (Bash only needs a function to exist when it is called), variables stay local to functions instead of leaking everywhere, and, as you will see in the testing section, other files can source the script to test its functions without running it.

Bash A script skeleton with main, usage and logging

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

A few details worth copying:

  • usage is a function that prints a here-doc, so -h (to stdout, exit 0) and "you used it wrong" (to stderr, exit 2) share the same text.
  • log writes to stderr. Programs that print results on stdout must keep diagnostics out of that stream, or the next stage of a pipeline will choke on them.
  • die prints a message prefixed with the program name and exits non-zero. Everything fatal goes through one function, so you can change the format or add a stack trace in one place.
  • parse_args sets global variables on purpose (they are the program's settings), while helpers use local for everything else.

Put readonly on constants (readonly BACKUP_DIR=/var/backups). It turns accidental reassignment into an immediate error, and it tells the next reader "this never changes".

Configuration: defaults, files, environment, flags

Real scripts have settings: a directory, a retention count, a timeout. Hard-coding them forces people to edit the script, which makes every deployment a fork. Give each setting a default and let it be overridden from outside, in a fixed order of priority (lowest to highest):

TEXT
   built-in default  <  config file  <  environment variable  <  command-line flag

That order matches what users expect from most Unix tools: a flag always wins, because it is the most specific and the most recent thing the user typed.

Bash Layered configuration

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The config file is parsed line by line instead of being sourced. Sourcing a file runs it as code, so anyone who can edit the config can run commands as the script's user. Reading KEY=value pairs and accepting only known keys is a few extra lines and removes that risk. printf -v "$key" '%s' "$value" assigns to the variable whose name is in $key without eval.

Dry-run: look before you leap

Any script that changes things (deletes, moves, uploads) should offer a dry-run flag that prints what it would do. The cleanest way is to route every side effect through one function:

Bash A run() wrapper for --dry-run

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

printf ' %q' prints each argument quoted so that you could paste the line into a shell, which also shows you exactly where spaces in file names would have caused trouble. The discipline is simple: never call mv, rm, cp or curl directly for something that matters; call run mv .... Then --dry-run cannot forget a step.

Idempotency and temporary files

A maintainable script is safe to run twice. Exit Codes, set -e and Error Handling lists the building blocks (mkdir -p, rm -f, ln -sfn, check before append). Two more rules complete the picture:

  • Create temporary files with mktemp, never with a fixed name like /tmp/out.txt. Fixed names collide when two copies run at once and can be hijacked on shared machines.
  • Write, then rename. Produce the new file next to its destination under a temporary name, then mv it into place. A rename within one filesystem is atomic, so readers see either the old file or the new one, never half of one.
Bash Atomic update with a temp file and a RETURN trap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The RETURN trap runs when the function returns, on both paths, so the half-written file never survives. In a whole script you would use trap ... EXIT instead (see Signals and trap).

Locking: only one copy at a time

A nightly job that takes longer than expected can overlap with the next run and corrupt shared data. The classic fix is a lock. Creating a directory is an atomic operation: mkdir either creates it or fails because it exists, with no gap between "check" and "create" in which another process could slip. That makes mkdir a portable lock.

Bash A mkdir lock that cleans up after itself

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Exit code 75 is EX_TEMPFAIL from the BSD sysexits.h convention ("try again later"); cron wrappers can treat it differently from a real failure. On Linux there is also the flock command, which ties the lock to an open file descriptor, so the kernel releases it automatically even if the script is killed with kill -9. A mkdir lock left behind by a killed script (a stale lock) needs manual cleanup, which is the price of its portability. Advanced Patterns shows flock in detail.

Testing a script

Scripts without tests break silently. Testing Bash is easier than people think, if the script follows the shape from the start of this lesson. The trick is to make the final line conditional:

Terminal
# only run main when executed, not when sourced
if [[ ${BASH_SOURCE[0]} == "$0" ]]; then
  main "$@"
fi

When another script does source tool.sh, $0 is the other script's name, so the condition is false and nothing runs, but every function is now available for testing.

You do not need a framework to start. A test runner is a function that compares an expected value with an actual one and counts failures:

Bash A hand-written test runner (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The runner ends with (( fail == 0 )), so its exit status is 0 only when everything passed and CI systems can use it directly. For bigger suites the community tool bats (Bash Automated Testing System) gives you @test "name" { ... } blocks and nicer reports. It is not installed in our sandbox, so this is a read-only example of its style:

Terminal
# test/slugify.bats  (needs bats; not runnable here)
setup() { source ./lib.sh; }

@test "slugify lowercases and joins words" {
  run slugify "Hello World"
  [ "$status" -eq 0 ]
  [ "$output" = "hello-world" ]
}

You will build a complete framework of your own in Project: A Mini Test Framework.

Logging and exit codes you can rely on

Pick a few log levels and stick to them. A common minimum is info, warn and error, all to stderr, each with a timestamp when the script runs unattended. Document your exit codes in the usage text if other programs call yours: 0 success, 1 runtime failure, 2 bad usage, and a few specific ones (like 75 above) where a caller must react differently.

Knowing when to stop

Bash is excellent glue: it runs programs, wires pipes and handles files. It is a poor general-purpose language. Consider rewriting in Python, Go, Rust or Node.js when you notice any of these:

  • the script passes a few hundred lines and still grows;
  • you need real data structures (nested objects, sets of records) or parse JSON, YAML or XML in anger;
  • you need serious error handling, unit tests with mocks, or concurrency beyond & and wait;
  • performance matters, because a loop of external commands is thousands of times slower than the same logic in a compiled or bytecode language;
  • Windows support is required.

None of that makes the script wrong. Shell is the right tool for starting programs and gluing their output together; the moment you are mostly manipulating data inside the script, a different tool serves you better. For a related set of pitfalls, see Performance, Tricks and Pitfalls.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why do well-structured scripts put all logic in functions and end with a single main "$@" line?

    Only the last line does anything. Functions need to exist only when called, variables can be local, and a guard around main lets tests source the file without running it.
  2. 2. In the layered configuration model, which source has the highest priority?

    From lowest to highest: built-in default, config file, environment variable, flag. The flag is the most specific and most recent choice by the user.
  3. 3. Why is mkdir lockdir a good way to take a lock?

    Directory creation is one atomic step, with no gap between checking and creating. You still need a trap to remove the lock, and a killed script can leave a stale one.
  4. 4. What is the purpose of [[ ${BASH_SOURCE[0]} == "$0" ]] && main "$@" at the end of a script?

    When a file is sourced, $0 is the name of the sourcing script, so the two values differ and main does not run. That lets tests load the functions safely.
  5. 5. What does a dry-run mode that routes every side effect through a run function achieve?

    When mv, rm and similar commands are only ever called via run, one if statement decides whether to print or execute, so --dry-run covers every action.

Summary

  • Structure scripts as: shebang, strict mode, constants, helpers (usage, log, die), one function per job, main, and a final main "$@".
  • Send diagnostics to stderr and results to stdout; exit non-zero on failure and document your codes.
  • Layer configuration as default, config file, environment, flag. Parse config files instead of sourcing them.
  • Route side effects through a run function to get a reliable --dry-run.
  • Use mktemp, write-then-rename updates and traps so interrupted runs leave nothing half-done; use a mkdir or flock lock to prevent overlapping runs.
  • Guard main so the file can be sourced, then test functions with a small assert runner (or bats).
  • Switch to a general-purpose language when a script grows into a program.

Next: Text Processing Recipes