Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 95 ยท Shell Scripting

Quoting and Expansion

Master Bash quoting: single vs double quotes, backslash, $'...', word splitting and globbing, "$@" vs $*, the order of expansions, and the spaces-in-filenames bug.

Quoting is the part of shell scripting that decides whether a script works with your test files or with every file. Almost every Bash bug that experienced people warn about, the script that deletes the wrong folder, the loop that breaks on a filename with a space, the variable that suddenly contains a list of files, is a quoting bug.

The reason is how the shell works. Before it runs any command, it performs a series of expansions on the line you typed: variables become their values, * becomes filenames, $(...) becomes output. Quotes are your control panel for those expansions: they say which ones are allowed and which are not. Once you know the rules, there are only a handful of them, and they become automatic.

Why the shell needs quoting

Consider the line rm $file. Bash does not see a "command with one file argument". It sees the characters, expands $file to its text, then splits that text at spaces and expands wildcards, and only then calls rm with the resulting words. If file holds my notes.txt, rm receives two arguments, my and notes.txt.

TEXT
   typed:            rm $file             file="my notes.txt"
   after expansion:  rm my notes.txt      <- two words, wrong!
   typed:            rm "$file"
   after expansion:  rm "my notes.txt"    <- one word, right
Bash The classic bug, shown safely

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In the first case ls complains ls: cannot access 'my': No such file or directory and also lists notes.txt, which is the wrong file. With rm instead of ls, you would have lost data. The printf trick at the end is a safe way to see exactly which words a command would receive.

The three quoting mechanisms

Bash has three ways to protect characters from the shell:

Mechanism Example Effect
Single quotes 'text' Everything is literal. Nothing is expanded.
Double quotes "text" Protects spaces and wildcards. Still expands $var, $(cmd), $((n)), and \ before a few characters.
Backslash \* Protects the single next character.

Single quotes: everything is literal

Inside '...' every character means itself, including $, \ and ". The only character you cannot have inside is the single quote itself.

Bash Single quotes keep everything literal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Use single quotes whenever you do not want any expansion: regular expressions, awk and sed programs, grep patterns.

Double quotes: expand, but do not split

Inside "..." the shell still expands $name, ${name}, $(command), $((1+2)), and backquotes. But the results are not split into words and wildcards are not expanded. Backslash keeps its special meaning only before $, `, ", \ and a newline.

Bash Double quotes expand but do not split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The nesting in the last line works because everything inside $( ... ) is its own quoting context, a feature of $( ) that backquotes do not offer.

Backslash: escape one character

A backslash outside quotes removes the special meaning of the next character. \ is a literal space, \* a literal star, and a backslash at the end of a line joins it with the next line.

Bash Backslash escapes one character

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

ANSI-C quoting: $'...'

For control characters (tab, newline) use $'...'. It is a Bash feature (also in zsh, ksh) that understands backslash escapes such as \n, \t, \\, \', \xHH and \uHHHH.

Bash $'...' for tabs and newlines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Note that echo -e and echo "a\nb" behave differently between shells and systems; prefer printf or $'...' for escapes, as explained in Creating and Viewing Files.

Word splitting and globbing of unquoted values

When an expansion (a variable, $(...), $((...))) is unquoted, the shell post-processes the result in two steps:

  1. Word splitting: the result is cut into words wherever it contains a character from IFS (by default: space, tab, newline). Runs of separators count as one.
  2. Pathname expansion (globbing): every word that contains *, ? or [...] is replaced by the sorted list of matching files, if any match.
Bash Splitting and globbing step by step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

If no file matches a wildcard, Bash leaves the pattern as it is (unless nullglob is set, see Wildcards and Globbing). That is why echo $pattern sometimes prints *.log and sometimes a list.

The rule of thumb: put double quotes around every variable and command substitution: "$var", "$(cmd)". The few times you want splitting, for example to build a list from a short string of words, say so deliberately in a comment.

"$@" versus "$*"

Both expand to all positional parameters, but differently:

  • "$@" expands to one word per argument, exactly as the caller gave them. It is almost always what you want.
  • "$*" expands to one single word, joined with the first character of IFS (a space by default).
  • Unquoted $@ and $* are identical and are split like any other variable. They break arguments that contain spaces.
Bash Passing arguments through

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Use "$@" whenever one function or script forwards its arguments to another command: ls -l "$@".

The order of expansions

When Bash reads a command it performs these steps, in this order:

TEXT
 1. brace expansion        {a,b}  {1..3}
 2. tilde expansion        ~  ~user
 3. parameter expansion    $var  ${var:-x}      \  left to right,
 4. command substitution   $(cmd)                >  all in one step
 5. arithmetic expansion   $((1+2))             /
 6. word splitting         only on UNQUOTED results of 3-5
 7. pathname expansion     * ? [..]             only on unquoted words
 8. quote removal          the quote characters themselves vanish

Two consequences are easy to demonstrate. First, brace expansion happens before variables are expanded, so a variable cannot be used inside {1..n}. Second, quote removal is last, so the shell knows about the quotes while the earlier steps run and the program never sees them.

Bash Why {1..$n} does not work

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

You will see all eight steps with more side-by-side comparisons in The Order of Expansions in Depth.

A quoting checklist

  1. Quote variables and substitutions with double quotes: "$file", "$(pwd)", "${arr[@]}".
  2. Use single quotes for text that must stay exactly as written: patterns, awk programs, sed scripts, trap commands.
  3. Do not quote what must expand: a glob you want expanded (*.txt), a tilde (~/bin), and the right side of [[ $x == pattern* ]] when you want pattern matching.
  4. Use -- before filenames so that a name that starts with - is not read as an option: rm -- "$file".
  5. Start a pipeline from find -print0 (and xargs -0) when filenames can contain anything.
  6. Test with printf '<%s>\n' ... to see where the word boundaries really are.
  7. Run shellcheck on your scripts: it flags unquoted variables (warning SC2086). It is not installed in our sandbox, but it is available for every Linux distribution.
Bash Hostile filenames

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In variant 3, rm -dash.txt makes rm read -d, -a, -s... as options and it stops with rm: invalid option -- 'a'; rm -- -dash.txt works. Variant 1 prints five entries (one filename even contains a newline, so the output of that entry spans two lines) and every name arrives whole. Variant 2 breaks the names with spaces and the one with a newline into pieces. The ./* form is also safe against names that start with a dash, since ./-dash.txt cannot look like an option.

The most dangerous unquoted variable is the one in rm -rf $dir/*. If dir is empty or unset, the command becomes rm -rf /*. Always quote and guard: rm -rf -- "${dir:?dir is not set}"/*. The ${var:?message} form aborts if the variable is unset or empty (Parameter Expansion and String Handling).

When not to quote

There are a few places where quotes would be wrong or are unnecessary:

  • The right side of an assignment needs no quotes for a single expansion: name=$other is safe (no word splitting in assignments), but quoting does no harm.
  • Inside [[ ... ]] the left side is not split either, but quotes are still good style. On the right side, unquoted means pattern or regex, quoted means literal text.
  • Inside $(( )) and (( )) names need no $ and no quotes.
  • case "$x" in does not split, though quoting is conventional.
Bash Quoting on the right of [[ ]]

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. file="my notes.txt" and you run rm $file. What does rm receive?

    Unquoted expansion results are split at whitespace. Writing rm "$file" keeps the name in one piece.
  2. 2. What does echo '$HOME' print?

    Single quotes make everything literal, including the dollar sign.
  3. 3. In a function called as f "a b" c, which form passes the arguments on unchanged?

    "$@" expands to one word per original argument. "$*" joins them into a single word, and the unquoted forms re-split them.
  4. 4. Why does echo {1..$n} not count up to n?

    The expansion order puts braces first and parameters later. Use seq or a C-style for loop when the bound is a variable.
  5. 5. Which character sequence gives you a real tab in Bash?

    Only the ANSI-C form $'...' interprets backslash escapes like \t and \n. Plain single and double quotes keep the backslash.

Summary

  • Before running a command, the shell expands it; quoting controls which expansions happen.
  • Single quotes: fully literal. Double quotes: allow $ expansions but stop word splitting and globbing. Backslash: escapes one character. $'...': C-style escapes.
  • Unquoted expansion results are split on IFS and then globbed. Quote "$var" and "$(cmd)" by default.
  • "$@" keeps arguments intact; "$*" joins them; unquoted forms break on spaces.
  • Order: braces, tilde, parameters/commands/arithmetic, word splitting, globbing, quote removal. That is why {1..$n} fails.
  • Protect against odd filenames with --, ./* globs and find -print0 | xargs -0; never parse ls.
  • Test word boundaries with printf '<%s>\n' ... and let shellcheck find the rest.

Next: Parameter Expansion and String Handling