Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 95 ยท Pipes and Redirection

xargs, Command Substitution and Process Substitution

Turn output into arguments with $( ) and xargs (-n, -I, -0, -P), and feed commands like files with process substitution <( ). Safe patterns with find -print0.

A pipe sends the output of one command to the input of the next. But many commands do not read their input at all: rm, mkdir, cp, chmod and ls take their work from arguments. You cannot write find . -name '*.tmp' | rm, because rm ignores stdin. Something has to turn text into arguments. In Bash there are three tools for the job, and they are all in this lesson.

  • Command substitution, $( ... ), replaces itself with the output of a command, so you can use that output inside another command line.
  • xargs reads items from stdin and runs a command with those items as arguments.
  • Process substitution, <( ... ), hands a command's output to another command as if it were a file, which is how you diff two pipelines.

Used carelessly, all three break on filenames with spaces. Used with the patterns below, they are safe and very fast.

Command substitution: $( )

$(command) runs command, captures what it writes to stdout, and puts that text on the command line in place of the $( ). The shell does this before the outer command starts.

TEXT
   echo "Today has $(wc -l < file) lines"
                    \_____________/
                     runs first, prints 3
   echo "Today has 3 lines"      <- what echo actually receives
Bash Command substitution basics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Why $( ) and not backticks

The old syntax is `command`. It does the same job, but nesting needs ugly escaping ( `echo \`date\ ``), backslashes inside behave oddly, and a backtick is easy to confuse with a quote. Always write $( ) in new code. You will meet backticks in old scripts, so you should be able to read them.

Quote it, or it gets split

The result of an unquoted $( ) goes through word splitting and globbing, exactly like an unquoted variable. Quote it unless you really want separate words:

Bash Unquoted substitution is split into words

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

$(cat file) is almost always avoidable. Write $(< file) (Bash reads the file without starting cat), or better, let the command read the file directly. Also remember that command substitution runs in a subshell, so assignments made inside it do not survive. See Subshells, Groups and Environment.

xargs: build command lines from input

xargs reads whitespace-separated items from stdin and runs a command with them appended as arguments. With no command it uses echo. Think of it as a bridge between "text on a pipe" and "arguments".

TEXT
 printf 'a b c' | xargs touch
                  items: a b c
                  runs:  touch a b c
Bash xargs turns input into arguments

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

When the list is long, xargs splits it into several runs so that no single command line exceeds the operating system's limit on argument size. On Linux that limit is typically about two megabytes, which is why rm *.log can fail with "Argument list too long" in a folder with a million files, but find ... | xargs rm does not.

-n: how many items per run

-n N caps the number of arguments per command. This is very useful for tools that accept only one input at a time.

Bash xargs -n: items per command

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first command prints four lines, because seven items in groups of two make batches of 2, 2, 2 and 1 items.

-I: put the item where you want it

By default the items go at the end of the command. With -I {} (the string {} is your choice) xargs runs the command once per input line and replaces every {} with that line. Spaces inside the line are kept, because with -I only newlines separate items.

Bash xargs -I places the item anywhere

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Use -I when the argument must appear in the middle, or appear more than once. It is slower than batching, because it starts one process per line.

The filename problem and -0

By default xargs splits on any whitespace and also interprets quotes and backslashes. So a file called my notes.txt becomes two items, my and notes.txt. This is the number one xargs bug:

Bash Spaces break plain xargs (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Real output: ls: cannot access 'my': No such file or directory and the same for notes.txt. The fix is to use the one character that cannot occur in a filename, the NUL byte (character zero). Unix forbids NUL in names, so it is a perfect separator. find -print0 writes names separated by NUL and xargs -0 reads them:

Bash find -print0 with xargs -0

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Always pair them: -print0 on the producer, -0 on xargs. Two more safeties are worth the typing: -- before filenames, so a name that starts with - is not read as an option, and -r (--no-run-if-empty), explained next.

Never feed ls output to xargs, for or any other command that parses it. Names can contain spaces, quotes and even newlines. Use find ... -print0, or a glob like for f in *.txt, see Wildcards and Globbing.

-r: do nothing when there is no input

GNU xargs runs the command once even when the input is empty. That is dangerous for commands that behave differently without arguments (rm complains, ls lists the current folder, grep waits for stdin). Add -r:

Bash xargs -r skips empty input

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

macOS and BSD xargs behave the other way round (they never run on empty input) and do not have -r. In portable scripts, avoid depending on this difference. See POSIX sh, Portability and Compatibility.

-P: run jobs in parallel

-P N runs up to N commands at the same time. Combined with -n 1 or -I, it spreads independent jobs over several processes. Our sandbox has one CPU, so you will not be faster here, but the structure is what you use on a real machine. Because the jobs run concurrently, their output may be interleaved and their order is not guaranteed, so sort it or write results to separate files.

Bash xargs -P runs commands concurrently

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The status line shows a value worth remembering: if any invocation exits with status 1 to 125, xargs itself exits with 123. (255 means a command was stopped by a signal or exited with 255, and xargs stops immediately.)

Other useful options

Option Effect
-t Print each command to stderr before running it (a cheap "verbose" mode)
-p Ask for confirmation before each command (interactive, not usable here)
-d '\n' Use a custom delimiter: GNU extension, handy for spaces in names when the input has one item per line
-L N Take N input lines per command
-s SIZE Maximum length of the generated command line
-a FILE Read items from FILE instead of stdin
--show-limits Print the limits of your system
Bash xargs -t and -d

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

xargs or find -exec?

find can run commands itself: -exec cmd {} \; runs once per file, and -exec cmd {} + batches many files into one command, exactly like xargs. Both are safe with odd filenames. Prefer -exec ... + for simple cases and xargs when you need -P, filtering with another command in between, or -I. See Finding Files.

Bash find -exec + versus xargs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Process substitution: <( ) and >( )

Some commands need files, not streams: diff a b, comm a b, paste a b, join a b. What if the two things you want to compare are the output of commands? You could write temporary files and clean up. Process substitution does it for you:

TEXT
   diff <(sort one.txt) <(sort two.txt)
         \___________/   \___________/
          runs, output     runs, output
          appears at       appears at
          /dev/fd/63       /dev/fd/62
   diff /dev/fd/63 /dev/fd/62       <- what diff receives

<(command) is replaced by a file name (usually /dev/fd/N) from which the command's output can be read. The program sees an ordinary path. It is a Bash (and zsh, ksh) feature, not POSIX sh.

Bash Compare two sorted views with diff

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first command shows the path with its number replaced, because the descriptor number can differ between runs.

The loop that keeps its variables

A very common use is feeding a while read loop. A pipe puts the loop in a subshell, so variables set inside it vanish when the loop ends. With < <(command) (a redirect from a process substitution) the loop runs in your shell:

Bash while read without a subshell

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Plain done < scores.txt is even simpler when the data is already in a file. Use < <( ) when the data comes from a command. Bash 4.2 and later also offer shopt -s lastpipe, which keeps the last pipeline stage in the current shell, but it only works when job control is off (as in scripts).

>( ): writing into a process

>(command) is the mirror image. It becomes a file name that the outer command can write to, and whatever it writes is fed to command's stdin. It lets one program's output go to several consumers at once:

Bash Split one stream with tee and process substitution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

There is a catch: the shell does not wait for the >( ) processes to finish. That is why the example sleeps briefly before reading the results. For anything real, prefer an explicit wait or design the flow so that the consumers write to files you read afterwards with a synchronization step. See Advanced Patterns for FIFOs and coprocesses.

Limits and portability

  • Process substitution needs a system that supports /dev/fd or named pipes. Linux and macOS do.
  • The file is a pipe, not a regular file. It cannot be read twice or seeked. A program that opens the file twice (some tools do, to check the size first) will fail or see no data. If so, write a temporary file with mktemp.
  • It is not available when Bash runs as sh in POSIX mode, nor in dash.

Choosing the right tool

Situation Best choice
Need command output as part of an argument or in a variable $( ), quoted
Many files from find for one command find ... -exec cmd {} +
Need parallelism, or a filter between find and the command find ... -print0 | xargs -0 -r -P N cmd
The command needs the item in the middle of its arguments xargs -I {} (or a while read loop)
A command wants two files but you have two pipelines <( )
Loop over command output and keep the variables while read ... done < <(cmd)
Complex per-item logic A while IFS= read -r -d '' f loop over find -print0

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does find . -name '*.tmp' | rm not delete anything?

    Pipes connect stdout to stdin. rm reads its arguments from the command line, so something such as xargs, find -exec or find -delete must supply them.
  2. 2. What is the safest way to pass the files found by find to grep -l pattern?

    -print0 and -0 separate names with the NUL byte, which can never appear in a filename. The other three break on spaces, quotes or newlines in names.
  3. 3. What does echo "$(printf 'a\n\n\n')" print, apart from the final newline echo adds?

    Command substitution strips all trailing newlines from the captured output, so only a remains.
  4. 4. A script runs ... | xargs rm and sometimes complains that rm has no operand. Which option avoids running rm on empty input (GNU xargs)?

    -r (--no-run-if-empty) tells GNU xargs not to run the command at all when it reads no items.
  5. 5. What does diff <(sort a.txt) <(sort b.txt) do?

    Process substitution runs each command and gives diff a path from which that command's output can be read, so no temporary file is needed.

Summary

  • $(cmd) replaces itself with the output of cmd (trailing newlines removed). Prefer it to backticks, and quote it.
  • xargs turns stdin items into arguments. It splits on whitespace by default, which breaks on odd filenames.
  • Use find ... -print0 | xargs -0 for safe file lists, -r to skip empty input, -n to batch, -I {} to place items, -P for parallel jobs.
  • find -exec cmd {} + is the simplest alternative to xargs for batches.
  • <(cmd) presents the output of a command as a file; >(cmd) presents a command's input as a file. They are Bash features, not POSIX sh.
  • done < <(cmd) keeps loop variables in the current shell; a pipe would run the loop in a subshell.

Next: Subshells, Groups and Environment