Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 95 ยท Shell Scripting

Debugging Shell Scripts

Debug Bash scripts like a pro: set -x and PS4 tracing, bash -n syntax checks, set -v, LINENO and FUNCNAME, trap DEBUG, ShellCheck warnings and assert helpers.

Every script has bugs. The shell does not give you a debugger with breakpoints, but it does give you something almost as good: the ability to print every command after expansion just before it runs. Most Bash bugs are about what the shell turned your text into (an unquoted variable, a glob that matched nothing, an empty value), and seeing the expanded command shows you the problem immediately.

In this lesson you will check syntax without running anything (bash -n), trace execution with set -x and make the trace readable with PS4, use LINENO, FUNCNAME and BASH_SOURCE to report where you are, step through code with trap DEBUG, learn the warnings ShellCheck gives (it is a read-only topic here), and write a tiny assert helper for testing scripts. Related lessons: Exit Codes, set -e and Error Handling and Quoting and Expansion.

Where trace output goes. set -x writes to standard error. In our editor stdout and stderr appear as two separate blocks, so in the examples below you will often see 2>&1 appended to the command that runs the script, which merges them in the right order.

The cheapest debugging tool: print things

Before any special option, remember the basics. Print the value of a variable with brackets around it so that empty values and stray spaces are visible, and write debug output to stderr so that it does not pollute real output.

Bash Brackets reveal empty values and spaces

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The debug function prints only when the DEBUG variable is set, so you can leave the calls in the script and switch them on with DEBUG=1 ./script.sh. The return 0 makes sure a hidden message does not return a failure status, which matters under set -e.

Check the syntax with bash -n

bash -n script.sh reads the script and parses it, but runs nothing. It catches missing fi, done and esac, unbalanced quotes and brackets. It cannot catch runtime problems like a misspelled command or variable.

Bash bash -n finds syntax errors without running (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

good.sh passes silently with status 0. The other two print a syntax error message with the line number (for example unexpected end of file at the line after the last one) and exit with status 2, and nothing from echo was printed. Run bash -n in a pre-commit hook or CI step; it costs nothing.

Tracing with set -x

set -x (also bash -x script.sh) prints each command to stderr, after expansion, with a + in front. Nested expansions get more plus signs: ++ for a command substitution, +++ one level deeper.

Bash set -x shows commands after expansion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Read the trace: you see the real argument list ls a.txt b.txt that $files expanded into, the loop variable taking each value, and the final echo. The lines without a plus are the script's own output. Because the trace and the output are interleaved, you can see which command produced which output.

Turning tracing on for part of a script

You do not have to trace the whole script. Put set -x before the suspicious part and set +x after it. (A leading + switches an option off, a historical oddity.)

Bash Tracing just one section

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice that the line set +x itself is printed (it runs while tracing is still on), but nothing after it. You can also enable tracing for a whole run without editing the file: bash -x script.sh, or BASH_XTRACEFD/SHELLOPTS=xtrace in the environment to trace a script and everything it launches.

A better trace with PS4

The + prefix comes from the variable PS4. Bash expands it before printing each traced line, so you can put useful information in it: the script name, the line number, the function name.

Bash PS4 with file, line and function name

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Every line now says where it came from. The key pieces:

  • LINENO: the number of the line being executed.
  • BASH_SOURCE: the file of the current code (useful when you source other files).
  • FUNCNAME: an array with the names of the active functions, the innermost first. FUNCNAME[0] is empty at top level, so the example uses ${...:-main}.

The PS4 value must be in single quotes, so the variables are expanded at trace time and not when you assign it. The trace of a command substitution shows ++-style repeated first character, which is why PS4's first character is conventionally a +.

set -v: echo the source lines

set -v (verbose) prints each input line as it is read, before expansion. Use it to see the script text as the parser sees it. Combined with -x you get both: the source line and the expanded command.

Bash set -v versus set -x

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In practice -x is the one you reach for 95% of the time. -v helps when a here-document or a multi-line construct does not parse the way you expected.

Where am I? LINENO, FUNCNAME, BASH_SOURCE and caller

Inside the script you can ask the shell where you are. A function that prints a stack trace is a handy thing to call from an error handler.

Bash Printing a call stack from an error handler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

FUNCNAME[i] is a function name, and BASH_LINENO[i-1] is the line inside that function from which the next-inner function was called. So "in inner(), line 11" means that inner called stack_trace on line 11, and "in main(), line 16" means the top-level code called outer on line 16. The builtin caller prints the same kind of information (caller 0 gives line function file). Combine this with trap 'stack_trace' ERR (Signals and trap) for error reports that show how the script got to the failure.

Stepping with trap DEBUG

The special trap DEBUG runs before every simple command. It is the closest thing the shell has to single-stepping. You can print the command about to run ($BASH_COMMAND), or make a poor man's debugger that pauses.

Bash trap DEBUG prints each command before it runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

An interactive stepper would read a key in the trap (read -r -n1 -p "step? "), but that needs a terminal, so you cannot try it here. The DEBUG trap slows a script down a lot; use it only while debugging and remove it afterwards.

Common bugs and what the trace shows

Most real-world failures fall into a few families. set -x makes each of them visible.

Bash Three classic bugs, seen through set -x

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In the trace:

  1. ls my file.txt: the shell passed two arguments. The fix is ls "$f" (Quoting).
  2. echo '*.nomatch': the pattern stayed as literal text because nothing matched. With shopt -s nullglob it would expand to nothing (Wildcards).
  3. name = Ada: the trace shows a command name run with arguments = and Ada; assignments cannot have spaces around =.

A fourth family is wrong exit status handling: the trace shows false or grep failing quietly while the script goes on. Use set -e or an explicit check, and look at $? in the trace.

ShellCheck (read-only)

ShellCheck is a static analyzer that reads a script and warns about the bugs above and many others, without running it. It is not available in our editor, so the snippets here are read-only. Install it with your package manager (apt install shellcheck, brew install shellcheck) and run shellcheck script.sh. Most editors can run it as you type.

Terminal
# bug.sh - read-only example
#!/bin/bash
files=$(ls *.txt)
for f in $files; do
  rm $f
done
cd /tmp/build
rm -rf *

ShellCheck would report warnings like these (codes shortened):

TEXT
SC2045: Iterating over ls output is fragile. Use globs.
SC2086: Double quote to prevent globbing and word splitting.
SC2164: Use 'cd ... || exit' in case cd fails.
SC2115: Use "${var:?}" to ensure this never expands to /*.

The warnings you will see most often:

Code Meaning Fix
SC2086 Unquoted variable Quote it: "$var"
SC2046 Unquoted command substitution "$(cmd)"
SC2164 cd without a failure check cd dir || exit 1
SC2162 read without -r read -r
SC2155 local x=$(cmd) hides the status Declare, then assign
SC2034 Variable assigned but never used Remove it or export it
SC2154 Variable used but never assigned Typo? Or set elsewhere
SC2016 $var inside single quotes will not expand Use double quotes if you want expansion
SC2006 Backticks are deprecated Use $(...)

When a warning is a false positive, silence it for one line with a comment: # shellcheck disable=SC2086, and explain why. Do not disable checks globally to make output quiet.

Testing your scripts: a tiny assert helper

Tracing finds bugs after you know they exist. Tests find them before. You do not need a framework for small scripts: a few functions that compare expected and actual values, count failures, and set the exit status are enough. Structure your script so that functions are testable, and do the real work only when run directly ([[ ${BASH_SOURCE[0]} == "$0" ]]).

Bash A small test runner with assert_eq

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The runner prints each failure with expected and actual values (in brackets) and the line of the failing assertion, and its exit status is non-zero when anything failed, so CI can use it. The check [[ ${BASH_SOURCE[0]} == "$0" ]] is true only when the file is executed directly; when another script sources it, BASH_SOURCE[0] is the library and $0 is the caller. For larger projects look at the bats framework (read-only here, not installed in our editor), or build on this idea in the "mini test framework" project later in the course.

A debugging checklist

  1. Read the error message carefully: Bash reports the script name and line (script.sh: line 12: ...).
  2. Run bash -n for syntax errors and ShellCheck for the usual mistakes.
  3. Run with bash -x (with a good PS4) and look at the expanded commands.
  4. Print suspicious variables in brackets.
  5. Shrink the problem: copy the failing lines into a new file with fixed inputs and try again.
  6. Check exit statuses: echo $? after the failing command; set -o pipefail for pipelines.
  7. Fix it, then add a test so it stays fixed.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does bash -n script.sh do?

    The -n option stands for noexec. Bash parses the file and reports syntax errors, but runs nothing. It cannot find runtime errors.
  2. 2. In the trace output of set -x, what do you see for the line ls $files when files="a.txt b.txt"?

    set -x prints commands after expansion. Because $files is unquoted, it is split into two words, and the trace shows both as separate arguments.
  3. 3. Which variable controls the prefix printed in front of each traced command?

    PS4 is the prompt for execution traces; its default is + . You can set it to include ${LINENO}, ${FUNCNAME[0]} and the like.
  4. 4. Why should you write export PS4='+ ${LINENO}: ' with single quotes?

    With double quotes ${LINENO} would be expanded once, when PS4 is assigned. Single quotes keep the text, and Bash expands it freshly for every traced command.
  5. 5. What does ShellCheck warn about in rm $file?

    SC2086 is the classic warning about unquoted variables: a file name with spaces or glob characters would be split or expanded. Write rm -- "$file".

Summary

  • Print variables inside brackets and send debug output to stderr; gate it with a DEBUG variable.
  • bash -n checks syntax without running; set -x (or bash -x) traces expanded commands to stderr; set +x turns it off; set -v echoes source lines.
  • Set PS4 in single quotes to include ${BASH_SOURCE}, ${LINENO} and ${FUNCNAME[0]} in every trace line.
  • FUNCNAME, BASH_LINENO and caller give you a call stack; trap '...' DEBUG runs before every command.
  • ShellCheck finds quoting, cd, read -r and many other problems before you run the script; silence a false positive with a # shellcheck disable= comment and a reason.
  • Small assert functions plus a non-zero exit status are enough to test script functions; guard the main code with [[ ${BASH_SOURCE[0]} == "$0" ]].

Next: Processes: ps, top and /proc