Skip to content
Tutorials Center
My learning

Topics

Lesson 64 of 95 ยท Advanced Bash

Performance, Tricks and Pitfalls

Make Bash scripts fast and safe: avoid forks and subshells, use builtins, replace slow while-read loops, run jobs in parallel, and dodge classic pitfalls like parsing ls.

Bash is not slow because it is a bad interpreter; it is slow because it makes you start programs. Every $(basename "$f"), every echo | grep, every cat file | wc -l creates a new process. On Linux that costs roughly a millisecond, which is nothing for ten files and everything for ten thousand. Most "my script takes forever" problems come down to counting forks.

This lesson shows where the time goes, how to avoid it, how to use more than one core, and then switches to the other half of shell expertise: the pitfalls that cause wrong results, not slow ones. It ends with the newer Bash 5 features that are worth knowing. You should be comfortable with Parameter Expansion, Loops and Subshells first.

Where the time goes: forks and execs

Running an external command costs two operations: the shell forks (copies itself) and the copy execs (loads the new program). Command substitution $(...) and every stage of a pipeline fork as well, even when they only run builtins.

TEXT
  result=$(basename "$f")          builtin: ${f##*/}
        |                                  |
   fork a subshell                  no new process,
        |                           just string handling
   exec /usr/bin/basename           inside the running shell
        |
   read its output through a pipe

The rule is simple: if the shell can do it with a builtin or parameter expansion, let it. Compare two ways of getting the file name from a path, 300 times. The script prints only whether the builtin was much faster, because exact timings differ between machines.

Bash External command versus parameter expansion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

On a typical machine the builtin version is hundreds of times faster, which is why the check uses a factor of 20 that leaves a wide safety margin. Useful builtin replacements:

Instead of Use
$(basename "$p") ${p##*/}
$(dirname "$p") ${p%/*}
$(echo "$s" | tr a-z A-Z) ${s^^}
$(echo "$s" | sed 's/a/b/g') ${s//a/b}
$(expr $a + $b), $(echo $a+$b | bc) $(( a + b ))
$(cat file) $(< file)
$(date +%s) $EPOCHSECONDS (Bash 5) or printf '%(%s)T'
$(wc -c <<< "$s") ${#s}

Note $(< file) is the one place where $() is cheap in Bash: it reads the file without running cat. Also note the real limit of this advice: ${var//a/b} is fast for short strings, but on megabytes of text sed is the better tool.

Avoid the subshell when returning values

A function that "returns" a value by printing it forces the caller to capture it with $(...), which forks. In a hot loop, assign to a variable instead. The two standard tricks are a global result variable and a nameref (local -n), which names the caller's variable:

Bash Returning values without a subshell

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Use this sparingly. It makes code harder to read, so reserve it for loops that run thousands of times; everywhere else, clean $(func) is fine.

Reading files line by line: when while read is the wrong tool

while IFS= read -r line is the right way to process lines when you need shell logic per line. But it is slow per line: the shell parses and runs the loop body each time. For simple per-line work on large files, one awk (or grep, sed, sort) process does the same work in a single pass.

Bash Summing a column: shell loop versus awk

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The two totals agree; only the cost differs. Two more rules for big files:

  • Do not read a whole file into a variable or array when you can stream it. grep and awk process gigabytes using almost no memory; mapfile on the same file needs all of it in RAM.
  • Avoid useless cat. grep pattern file instead of cat file | grep pattern saves a process and a copy. sort < file and wc -l < file do too.

And a reminder from Standard Streams and Redirection: a loop's redirect is done once. while ...; done < file opens the file one time, while a cat file or grep inside the loop body would start a process on every iteration.

Using more than one core

Shell scripts run commands one after another. Independent work can overlap with & and wait, or with xargs -P. Parallelism helps most when the work waits (network, disk) or is CPU-heavy on a multi-core machine. Our sandbox has one CPU, so this example uses sleep to stand for waiting jobs:

Bash Serial versus parallel with xargs -P

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

export -f makes the function visible to the child bash -c. The sort makes the output deterministic, because parallel jobs finish in whatever order the scheduler picks. When you run your own background jobs, cap the number running at once with wait -n (Bash 4.3+), which returns as soon as any one job finishes:

Bash A job pool limited to 2 workers with wait -n

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

See Jobs and Background Processes for the basics and Advanced Patterns for more.

The pitfalls checklist

Speed bugs are annoying; correctness bugs destroy data. These are the mistakes that cause most of them.

Parsing ls

ls is for humans. Its output is ambiguous when names contain spaces or newlines, and for f in $(ls) splits on whitespace. Use a glob instead:

Bash for f in $(ls) versus a glob

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

A failing cd

cd somewhere; rm -rf * deletes the current directory's contents if cd fails. Always check: cd "$dir" || exit 1, or write cd "$dir" && rm -rf -- ./*. In a script with set -e, a plain failing cd stops the script, but do not depend on that when the line is inside a condition or function used in one.

Unquoted variables and missing --

An unquoted $var is split into words and expanded as a glob, so a file named -rf or a b or * changes what a command does. Quote every expansion ("$var", "${arr[@]}") and put -- before names that may start with a dash (rm -- "$f"). Quoting and Expansion has the details.

Pipelines and variables

Each stage of a pipeline runs in a subshell, so variables set inside ... | while read ... vanish when the loop ends. Feed the loop with redirection or process substitution instead, or set shopt -s lastpipe:

Bash The vanishing counter in a pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Other regular offenders

  • [ $x = y ] fails when $x is empty or contains spaces. Use [[ $x == y ]] in Bash.
  • echo interprets some strings (-n, -e, backslashes) differently across shells. Use printf '%s\n' "$x".
  • cmd1 && cmd2 || cmd3 is not if/else: cmd3 also runs when cmd2 fails.
  • Numbers with a leading zero are octal in arithmetic: $((08)) is an error.
  • read without -r eats backslashes, and without IFS= trims whitespace.
  • set -e has holes (Exit Codes, set -e and Error Handling).
  • Run shellcheck on every script. It finds most of the items above automatically.

Bash 5 features worth knowing

Bash 5 added several conveniences. Our sandbox runs Bash 5.0, and a few things below need 5.1 or later (marked), so check $BASH_VERSION on your own system.

Bash Bash 5 conveniences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Worth a note, though not available in 5.0: Bash 5.1 added $SRANDOM (a 32-bit random number from the system), wait -p var (store the PID of the job that finished) and the ${var@U}, ${var@L}, ${var@u} case transforms. Bash 5.2 added patsub_replacement, where & in a ${var/pat/rep} replacement stands for the matched text. Do not use these in scripts that must run on older systems (macOS still ships Bash 3.2).

When to use what

  • Hot loops over thousands of items: builtins, parameter expansion, no $(...) per iteration.
  • Line-oriented text on big files: one awk/grep/sed, not a while read loop.
  • Independent slow jobs: xargs -P or & with a wait -n pool.
  • Everything else: write the clearest code first, measure with time, and optimize only what the numbers point at. A script that runs once a day and takes two seconds does not need tricks.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is name=${path##*/} much faster than name=$(basename "$path") in a loop?

    The expansion is handled inside the running shell. The basename version forks, execs a program, and reads its output through a pipe on every iteration.
  2. 2. What goes wrong with cat file | while read -r line; do count=$((count+1)); done followed by echo $count?

    Each pipeline stage runs in its own subshell, so changes to variables vanish when it ends. Use while ...; done < file or process substitution.
  3. 3. Which approach is best for summing one column of a 10-million-line file?

    awk streams the file in one process with compiled-in field splitting. A shell loop pays interpreter overhead per line, and the array or $(cat) approaches need the whole file in memory.
  4. 4. What does wait -n do?

    wait -n (Bash 4.3+) returns as soon as one background job ends, which makes it the building block for a limited-size job pool.
  5. 5. Why is for f in $(ls) considered a bug?

    The output of ls is split on whitespace and expanded as globs. A glob like for f in * hands each file name over intact.

Summary

  • Most shell slowness is process creation. Prefer builtins and parameter expansion over basename, dirname, tr, expr and date in loops.
  • $(...) and pipeline stages fork; return values through variables or namerefs only in hot loops.
  • Use one awk, grep or sed for line processing of large files rather than while read; stream instead of loading into memory.
  • Parallelize independent jobs with xargs -P or & plus wait -n.
  • Avoid the classic pitfalls: parsing ls, unchecked cd, unquoted variables, --, loops in pipelines, [ ] surprises and echo portability.
  • Bash 5 adds $EPOCHSECONDS, $EPOCHREALTIME, ${var@Q} and wait -n; 5.1 and later add $SRANDOM and wait -p.

Next: Project: A Backup Script