Skip to content
Tutorials Center
My learning

Topics

Lesson 58 of 95 ยท Advanced Bash

The Order of Expansions in Depth

Learn the exact order in which Bash expands a command line: braces, tilde, parameters, command substitution, word splitting, globbing, quote removal, and why eval bites.

Every Bash line goes through a fixed assembly line before a command runs. You met the pieces one at a time: globbing, variables, quoting, parameter expansion, arithmetic. This lesson puts them in order, because the order explains behaviour that otherwise looks random: why {1..$n} does not count to $n, why $cmd cannot contain a pipe, why unquoted variables suddenly become file lists, and why eval is the sharpest tool in the shed.

Once you can name the stage in which a surprise happens, you can fix it on purpose instead of by trial and error. The method of this lesson: a tiny helper that shows the words a command receives, and side-by-side examples where only one stage changes.

The assembly line

After Bash has split the line into words and recognised operators and redirections, it expands each word in these stages, in this order:

TEXT
  word as typed:   ~/{a,b}$suffix.txt

   1. brace expansion             {a,b}                -> several words
   2. tilde expansion             ~                    -> /home/ada
   3. parameter and variable      $suffix  ${x:-y}     \
      arithmetic expansion        $((1+2))              > same stage,
      command substitution        $(cmd)  `cmd`        /  left to right
      process substitution        <(cmd)  >(cmd)
   4. word splitting              only on UNQUOTED results of stage 3
   5. pathname expansion          * ? [..]  (globbing) on each resulting word
   6. quote removal               the remaining quotes and backslashes disappear

The three rules behind the whole lesson:

  1. Each stage sees the output of the stage before it, never the other way round. Braces are gone before variables exist; variables are expanded before the line is split into words.
  2. The result of an expansion is not scanned again for earlier stages. A variable holding {a,b} or $HOME or $(date) is not re-expanded. Text in a variable is data.
  3. Quoting turns stages off. Single quotes switch off everything. Double quotes keep stage 3 and switch off 1, 4 and 5 for that word (and tilde, braces).

(Interactive shells also do history expansion (!!) before all of this; it is off in scripts, so we ignore it.)

A helper to see the words

To watch the stages work we need to see what a command receives. This function prints the number of arguments, then each argument in brackets, so empty words, spaces and splits become visible (2: [a b] [c] means two arguments, a b and c):

Terminal
args() { printf '%d:' "$#"; printf ' [%s]' "$@"; printf '\n'; }

We will paste it at the top of each example, since every TryIt starts fresh.

Stage 1: braces come first, and know nothing about variables

Brace expansion is textual. It generates words from a pattern before any variable has a value. That is why a variable inside a sequence does not work:

Bash Braces are expanded before variables

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

{1..$n} is left as the literal text {1..3} (the variable expanded later, by which time braces were already finished). The reliable fixes are seq or an arithmetic for loop. The eval "echo {1..$n}" trick also works, and shows why eval is tempting and dangerous (below).

Stage 2: tilde

A leading unquoted ~ becomes $HOME; ~user becomes that user's home directory. Tilde is also expanded after a : or the first = in a variable assignment, which is what makes PATH=~/bin:~/tools:$PATH work. Quoting prevents it.

Bash Where tilde expands and where it does not

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last line is rule 2 in action: $t contains the characters ~/data, and tilde expansion (stage 2) already happened before $t (stage 3) was replaced. That surprises people who read a path from a config file or from read. If you really need it, expand the tilde yourself: ${t/#\~/$HOME}.

Stage 3: parameters, arithmetic and command substitution, left to right

These share a stage, so Bash simply works through the word from left to right. That makes side effects order-dependent:

Bash Side effects happen left to right

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last example is a classic trap. v=new cmd "$v" sets v only in the environment of cmd, but the argument "$v" is expanded by the shell before the assignment takes effect, so it still sees old. If you ever write IFS=, read ... <<< "$line", you rely on the opposite property: the assignment is in the environment of read itself.

Process substitution <(cmd) also lives here. It produces a filename (like /dev/fd/63) that connects to the command's output; see Building Text Pipelines.

Stage 4: word splitting, only on unquoted results

Word splitting happens only on the unquoted results of stage 3, never on literal text you typed, and it uses the characters in IFS (space, tab and newline by default). Empty unquoted results vanish completely; quoted ones stay as an empty word.

Bash What word splitting touches

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice that IFS=, also changes how $list is split but did not change how the literal a,b,c was parsed: the parser had already finished with it. (The default IFS also trims runs of spaces; a non-whitespace IFS like , makes every comma a boundary, and a,,c yields an empty word between.) Since Bash 4 the builtins declare, local and export also treat x=$v as an assignment and skip splitting; other shells do not. In portable scripts, always quote.

Stage 5: pathname expansion

After splitting, each word that contains an unquoted *, ? or [...] is replaced by the sorted list of matching paths, if any exist. Crucially this happens to split results of variables too, so one unquoted variable can trigger both stage 4 and stage 5:

Bash Splitting then globbing, and how to turn globbing off

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

That is why the old advice "quote your variables" is really "stop stages 4 and 5 from running on data". When you want splitting, such as turning a string into words, prefer read -r -a or turn globbing off with set -f in a subshell as shown.

The classic: expanding an unquoted command

Stage ordering also explains why you cannot store a pipeline or a quoted argument in a variable:

Bash A command in a variable only goes through stages 4 to 6

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The text 'big world' in the string was not parsed again as quoting, because quote handling happened when the line was first read. After expansion, the apostrophes are just characters and | is just an argument. Arrays hold words; strings hold text.

Stage 6: quote removal

Last, the quote characters that were part of your original syntax disappear. Quote characters that came out of an expansion are not removed (and not interpreted), since quote removal only applies to quotes in the original word.

Bash Quote removal and quotes from expansions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

eval: running the assembly line twice

eval takes its arguments, joins them with spaces, and then runs the entire line through every stage again. That is exactly why it is useful (it gives you a second pass in which new text is parsed) and why it is dangerous (anything in your data that looks like syntax becomes syntax).

Bash eval makes data into code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Treat eval as a last resort. Whenever user-controlled text (filenames, arguments, a config value) can reach it, you have created a command-injection hole, which is one of the most common serious bugs in shell scripts. Alternatives, in order of preference: an array of words; indirect expansion and namerefs (next section); printf -v name; and, only if you must, eval with every dynamic piece escaped by printf %q or ${var@Q}.

Indirect expansion and namerefs

Often the reason people reach for eval is "I have a variable name in another variable". Bash has direct tools for that:

Bash Variables whose names are in variables, without eval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

printf -v "$name" ... assigns to a computed variable name without evaluating the value, and local -n lets a function modify its caller's variable by name (see Functions and Arrays and Associative Arrays). Both still need the name to be trusted: validate it with [[ $name =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] if it comes from outside.

Arithmetic evaluates names and code, too

Arithmetic contexts ((( )), $(( )), array subscripts, -eq inside [[ ]]) treat a bare word as a variable name and evaluate it recursively, and an array subscript is expanded, including command substitutions. A "number" that you did not validate can therefore run commands:

Bash An unvalidated number is code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Seeing the stages: set -x

When you cannot tell which stage did what, turn on tracing. set -x prints each command after expansion, with quotes added where the words contain special characters, which is exactly the "words the command receives". Tracing goes to stderr, so we merge it with 2>&1 here:

Bash xtrace shows the result of all stages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Compare the two ls lines in the trace: the first, unquoted, shows the glob already replaced and b c.txt quoted as one word (globbing results are never re-split), the second shows what braces produced. For the whole toolbox, see Debugging Shell Scripts.

Practical consequences

  • Quote every expansion that holds data ("$var", "$(cmd)", "${arr[@]}") unless you are deliberately splitting.
  • Braces cannot count to a variable; use seq, for ((...)), or eval only with trusted numbers.
  • Never store a command with quoting or operators in a string; use arrays or functions.
  • Tilde and braces are not re-expanded in data from variables, files or read.
  • Never eval data you do not control, and validate anything that enters arithmetic.
  • Know which stage you are in when something looks wrong: add set -x and read the words.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does n=3; echo {1..$n} print?

    Brace expansion runs before variable expansion, so at that moment {1..$n} is not a valid sequence and stays as text. Then $n is expanded to 3, giving the literal {1..3}.
  2. 2. t='~/data' and then ls $t. Why does ls look for a directory literally called ~?

    Stages run in order, and tilde expansion (stage 2) comes before parameter expansion (stage 3). The tilde that arrives inside $t is just data.
  3. 3. In which order do these stages run: word splitting, pathname expansion, brace expansion, quote removal?

    Braces first, then tilde and parameter expansions, then word splitting, then globbing on each resulting word, and quote removal last.
  4. 4. cmd="echo 'a b' | wc -w"; $cmd fails to behave like the pipeline. What is the cleanest fix?

    After expansion, quotes and | in the string are ordinary characters, because the parser has already finished. Functions and arrays hold real shell structure.
  5. 5. Why is eval "echo $input" dangerous when input comes from a user?

    eval gives the whole line a second pass through parsing and expansion. Data containing ;, $(...) or backticks is then executed instead of printed.

Summary

  • Expansion order: braces, tilde, parameters / arithmetic / command and process substitution (left to right), word splitting, pathname expansion, quote removal.
  • Each stage works on the output of the previous one, and results of expansions are not scanned for earlier stages: variables hold data, not syntax.
  • Word splitting and globbing only act on unquoted results of expansions, which is why "$var" is the default.
  • Braces cannot see variables ({1..$n}), tilde is not expanded inside variables, and v=x cmd "$v" expands $v before the assignment.
  • Store commands in arrays or functions, not strings.
  • eval is a second full pass and an injection risk; prefer ${!name}, printf -v, local -n, associative arrays, and printf %q / ${var@Q} if you must.
  • Validate numbers before arithmetic, and use set -x to see the final words.

Next: Advanced Patterns: Regex, Process Substitution and Coprocesses