Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 95 ยท Shell Scripting

Loops: for, while and until

Learn Bash loops: for over lists, globs and ranges, C-style for, while and until, break and continue, safe file loops and reading lines with IFS= read -r.

Computers are good at doing the same thing many times. In the shell, that means loops: rename every file in a folder, process every line of a log, retry a command until it works, count from one to ten. Bash gives you three loop keywords (for, while and until) and a handful of habits that decide whether your loop is robust or quietly breaks on a file name with a space.

In this lesson you will learn each loop form, how break and continue steer them, how to loop over files safely, how to read a file line by line without losing data, and the pitfalls that professionals watch for: unmatched globs, parsing ls, the pipeline subshell trap and loops that eat their own input.

for loops over a list

The classic form walks through a list of words and runs the body once for each, assigning the current word to a variable:

TEXT
   for VAR in WORD1 WORD2 WORD3; do
       commands using "$VAR"
   done

The list can be written out, produced by brace expansion ({1..5}), a glob (*.txt), the positional arguments ("$@"), an array, or the result of a command substitution. Bash also has a C-style form with three arithmetic parts: for ((init; condition; step)).

Bash for loops: lists, ranges and C-style

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

What to take from this:

  • {10..0..5} counts down in steps of 5 (10 5 0). Brace expansion happens before variables are expanded, so it only works with literal numbers (more on that below).
  • The C-style loop accepts several variables, separated by commas, in each part. It is the natural choice when you need a counter and arithmetic (Arithmetic and Numbers).
  • for arg; do with no in loops over the positional parameters, exactly like for arg in "$@". After set -- we gave it three arguments, and "beta gamma" stayed one item because "$@" keeps each argument whole.
  • for x in $(echo "a b c") loops over the words of the command's output. That is fine for simple words but breaks on spaces and wildcards inside items (see the pitfalls below).
  • An empty list runs the body zero times. The variable keeps whatever value it had before (here, unset). After a loop finishes normally it keeps the last value assigned.

Always put double quotes around the loop variable when you use it: "$f". The quoting does not belong to the list but to every later use. See Quoting and Expansion.

Looping over files

Looping over files is the most common use of for in shell scripts, and the globs do all the work. The glob expands to the list of matching names, each one a single item, even if it contains spaces:

Bash Looping over files safely and unsafely

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Four lessons are packed in here:

  1. A glob that matches nothing stays as the literal text. for f in *.xyz ran once with f set to *.xyz. A script that then runs rm "$f" or wc -l "$f" operates on a file that does not exist. Fix it with shopt -s nullglob, which makes an unmatched glob expand to nothing (the loop runs zero times), or check inside the loop with [[ -e $f ]] || continue.
  2. Hidden files are skipped by *, unless you set shopt -s dotglob. (Even then . and .. are never included.)
  3. A directory is also a match for *. Filter with [[ -f $f ]] || continue or [[ -d $f ]].
  4. Never loop over $(ls ...). my notes.txt was split into two words, my and notes.txt. Word splitting cannot be fixed by quoting $(ls), because then the whole output becomes a single word. Use the glob directly. For recursive searches, use find with -print0 or -exec (Finding Files: find, locate and which) or shopt -s globstar and **/*.txt.

while and until

while repeats as long as a command succeeds (exit status 0). until is the mirror image: it repeats until a command succeeds. As with if, the condition is any command, so you can use [[ ]], (( )), grep -q, read and so on (see Conditionals).

TEXT
   while  COMMAND; do ... done      repeat while COMMAND returns 0
   until  COMMAND; do ... done      repeat while COMMAND returns non-zero

break, continue and loop status

break leaves the innermost loop. continue skips to the next iteration. Both take an optional number to affect an outer loop: break 2 leaves two levels, continue 2 continues the loop one level up.

Bash while, until, break and continue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Tracing the nested loops: for a=1 the body prints 1x, skips y (plain continue) and prints 1z. For a=2 the first b is x, and continue 2 jumps to the next value of a without printing. For a=3 it prints 3x, skips y, then at z the break 2 leaves both loops.

More facts shown here:

  • while true loops need a guard. An infinite loop is a legitimate tool (servers, retries) but every one needs a way out: a counter, a deadline, or a break condition. Our sandbox stops anything after 5 seconds, which is a good reminder to build the exit first. while :; do is the older spelling of while true; do.
  • The exit status of a for loop is the status of the last command executed in the body (false gives 1). A while loop that ends because its condition failed returns 0 (unless the body ran a failing last command).
  • A whole loop is a single command: you can pipe it into sort, redirect it into a file with > rows.txt (the file is opened once, not for each iteration) or run it in the background with &.
  • until [[ -s file ]]; do sleep 1; done is the standard "wait for something" loop. Add a timeout counter in real scripts.

Reading lines: while read

To process a file one line at a time, loop with read. The built-in read reads one line of standard input into a variable and returns success while there are lines left. Feed the loop with a redirect after done:

Terminal
while IFS= read -r line; do
  echo "$line"
done < file.txt

Two parts of that header are not optional if you care about correctness:

  • IFS= (set only for the read) stops read from stripping leading and trailing whitespace.
  • -r stops read from treating a backslash as an escape character that joins lines or eats itself.
Bash What read does without IFS= and -r

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The plain read trimmed the indentation off the second line and removed the backslash from back\slash, printing backslash. With IFS= read -r both lines come back exactly as stored. Set IFS to another character to split a line into several variables, as in the last line, where IFS=: splits on colons (and the extra { ...; } keeps the variables inside the same pipeline stage, see Subshells, Groups and Environment).

Bash Files, CSV fields and the missing final newline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Important details:

  • A last line without a trailing newline is dropped. read returns failure when it hits end-of-file before a newline, even though it did fill the variable. Adding || [[ -n $line ]] handles that. Text files from tools that do not end lines properly are common, so this guard is worth keeping in reusable scripts.
  • done < <(command) feeds the loop from a command without a pipe, so the loop runs in your current shell and total survives (42). With cat ... | while, the loop is in a subshell and the final total is back to 0. This is the number one reason variables "disappear" after a loop. Redirect from a file whenever you can.
  • The CSV trick IFS=, is fine for simple data. It does not understand quoted fields with commas inside. For those, use awk or a real CSV tool (awk: Column Processing and Reports).
  • The extra read before the inner loop consumes the header, because both read from the same redirected input.

A loop can steal its own input

Inside while read ... done < file, standard input is the file. Any command in the body that also reads stdin (cat, ssh, xargs, read) will swallow the rest of the file, and the loop ends after one iteration. The fix is to read the loop's input from a different file descriptor, so the body keeps the normal stdin:

Bash Commands in the body that eat stdin

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first loop printed only line one, because cat consumed two and three. In the second loop, 3< list.txt opens the file on descriptor 3 for the whole loop, and read -r l <&3 reads from it. The commands in the body are free to use stdin. This is also how you avoid ssh and ffmpeg swallowing your list (or give them < /dev/null). More on file descriptors in Standard Streams and Redirection.

Other ways to build a list

Not every list comes from a glob or a file. Some good options:

Bash find -print0, mapfile and seq

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

  • find -print0 with read -d '' is the bulletproof way to loop over file names of any kind, even with newlines in them. NUL is the only character that cannot appear in a path. (The sort -z only makes the order stable here.)
  • mapfile -t reads lines into an array, so you can loop over them later without worrying about a pipeline subshell. See Arrays and Associative Arrays.
  • {1..$n} does not work, because brace expansion runs before variable expansion: the loop ran once with the literal text {1..4}. Use seq, or better the C-style loop that needs no external program.
  • The list of a for is evaluated once, before the first iteration. Appending to the array inside the loop did not make the loop run longer.

When to use which loop

Need Use
Each of a few known items, or files for x in ... with a glob
A counter or index for ((i = 0; i < n; i++))
Lines of a file while IFS= read -r line; do ... done < file
Repeat until something happens (a retry, a wait) until command; do ...; done with a limit
Process very large inputs fast Not a shell loop: use awk, sed or grep, which are far quicker than running a Bash loop per line

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does for f in *.xyz; do echo "$f"; done print when no .xyz files exist (and nullglob is off)?

    Without nullglob an unmatched pattern is left as is, so the loop body runs once with f set to *.xyz. Use shopt -s nullglob or test with [[ -e $f ]].
  2. 2. Why is for f in $(ls) a bad way to loop over files?

    Command substitution output is word-split, so my notes.txt becomes two items. Use a glob (for f in *) or find -print0.
  3. 3. What are IFS= and -r for in while IFS= read -r line?

    With an empty IFS read does not trim whitespace; with -r it does not interpret backslash escapes. Together they return each line exactly as stored.
  4. 4. After total=0; cat f | while read -r x; do total=$((total+1)); done, what is the usual surprise?

    Each stage of a pipeline is a separate process, so changes to variables inside the loop are lost. Use done < f or done < <(cmd).
  5. 5. What does break 2 do inside two nested loops?

    break N leaves N enclosing loops. continue N similarly continues the Nth enclosing loop.

Summary

  • for x in list loops over words (globs, "$@", arrays, braces); for ((i = 0; i < n; i++)) is the counting form; while repeats while a command succeeds and until while it fails.
  • break [N] and continue [N] control the innermost or outer loops. Every potentially infinite loop needs a guard.
  • Loop over files with globs, not $(ls). Quote "$f", use nullglob for empty matches and dotglob for hidden files.
  • Read lines with while IFS= read -r line; do ... done < file; add || [[ -n $line ]] for a final line without newline.
  • A loop after a pipe runs in a subshell: redirect from a file or use < <(cmd) to keep variables. Commands that read stdin inside the loop can steal the input: use another file descriptor.
  • {1..$n} does not expand variables; use seq or the C-style loop. The list of a for is fixed when the loop starts.

Next: Functions