Linux Command Reference ยท Bash Syntax
Bash pipelines and pipefail
Reference for Bash pipelines: connect commands with |, understand exit status, set pipefail, read PIPESTATUS and avoid the subshell trap with variables.
A pipeline is a sequence of commands separated by |. The shell starts all commands at the same time and connects the standard output of each one to the standard input of the next, so data flows through without temporary files. Pipelines are the core of the Unix style: small tools that each do one job, combined on one line.
By default the exit status of a pipeline is the status of its last command, which hides failures earlier in the chain. The pipefail option and the PIPESTATUS array let you see them.
cmd1 | cmd2 | cmd3 cmd1 |& cmd2 # also pipe standard error ! cmd1 | cmd2 # negate the exit status
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.
Syntax
[time [-p]] [!] command1 [ | command2 ... ] [time [-p]] [!] command1 [ |& command2 ... ]
| Part | Description |
|---|---|
| |
Connect standard output of the left command to standard input of the right command. |
|& |
Shorthand for 2>&1 |: connect standard output and standard error. |
! |
Reverse the final exit status (0 becomes 1, anything else becomes 0). |
time |
Report elapsed time for the whole pipeline. |
Returns: the exit status of the last command, unless set -o pipefail is on: then it is the status of the rightmost command that failed, or 0 if all succeeded.
Usage notes
- All stages run concurrently, each in its own process (a subshell for compound commands). A fast producer blocks when the pipe buffer is full, so memory use stays small even with huge inputs.
- When a reader exits early (as
headdoes), the writer receives SIGPIPE and stops. That is normal; withpipefailit can make the pipeline report status 141 (128 + 13), which surprises scripts usingset -e. PIPESTATUSholds the status of every stage of the most recent foreground pipeline. Copy it immediately (s=("${PIPESTATUS[@]}")), because the next command overwrites it.- Variables set inside a pipeline stage are lost, because the stage runs in a subshell.
shopt -s lastpipe(effective when job control is off, as in scripts) runs the last stage in the current shell. A here-string or process substitution (done < <(cmd)) is the usual alternative. set -eonly looks at the final status, sofalse | truedoes not stop a script unlesspipefailis also set. Many strict scripts useset -euo pipefail.- Do not write
cat file | grep xwhengrep x fileworks; the extra process is harmless but needless. Pipelines are not seekable: tools that need to rewind a file cannot read a pipe. - A pipeline ending with
&runs in the background as one job. Stages cannot ask the terminal for input without confusing each other. - Because
|is a control operator, a newline right after it continues the pipeline, so long pipelines can be written one stage per line.