Skip to content
Tutorials Center
My learning

Topics

Linux Command Reference ยท Bash Syntax

Bash pipelines and pipefail

Reference for Bash pipelines: connect commands with |, understand exit status, set pipefail, read PIPESTATUS and avoid the subshell trap with variables.

A pipeline is a sequence of commands separated by |. The shell starts all commands at the same time and connects the standard output of each one to the standard input of the next, so data flows through without temporary files. Pipelines are the core of the Unix style: small tools that each do one job, combined on one line.

By default the exit status of a pipeline is the status of its last command, which hides failures earlier in the chain. The pipefail option and the PIPESTATUS array let you see them.

Terminal
cmd1 | cmd2 | cmd3
cmd1 |& cmd2        # also pipe standard error
! cmd1 | cmd2       # negate the exit status

Try it

Bash Building a pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Bash Exit status, pipefail and PIPESTATUS

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Syntax

Terminal
[time [-p]] [!] command1 [ | command2 ... ]
[time [-p]] [!] command1 [ |& command2 ... ]
Part Description
| Connect standard output of the left command to standard input of the right command.
|& Shorthand for 2>&1 |: connect standard output and standard error.
! Reverse the final exit status (0 becomes 1, anything else becomes 0).
time Report elapsed time for the whole pipeline.

Returns: the exit status of the last command, unless set -o pipefail is on: then it is the status of the rightmost command that failed, or 0 if all succeeded.

Usage notes

  • All stages run concurrently, each in its own process (a subshell for compound commands). A fast producer blocks when the pipe buffer is full, so memory use stays small even with huge inputs.
  • When a reader exits early (as head does), the writer receives SIGPIPE and stops. That is normal; with pipefail it can make the pipeline report status 141 (128 + 13), which surprises scripts using set -e.
  • PIPESTATUS holds the status of every stage of the most recent foreground pipeline. Copy it immediately (s=("${PIPESTATUS[@]}")), because the next command overwrites it.
  • Variables set inside a pipeline stage are lost, because the stage runs in a subshell. shopt -s lastpipe (effective when job control is off, as in scripts) runs the last stage in the current shell. A here-string or process substitution (done < <(cmd)) is the usual alternative.
  • set -e only looks at the final status, so false | true does not stop a script unless pipefail is also set. Many strict scripts use set -euo pipefail.
  • Do not write cat file | grep x when grep x file works; the extra process is harmless but needless. Pipelines are not seekable: tools that need to rewind a file cannot read a pipe.
  • A pipeline ending with & runs in the background as one job. Stages cannot ask the terminal for input without confusing each other.
  • Because | is a control operator, a newline right after it continues the pipeline, so long pipelines can be written one stage per line.