Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 95 ยท Pipes and Redirection

Pipes and Pipelines

Learn how Linux pipes connect processes: concurrency, pipefail and PIPESTATUS, tee, buffering, SIGPIPE with head, named pipes (mkfifo) and performance.

The pipe character | is the most famous symbol of the Unix shell. It connects the standard output of one command to the standard input of the next, so that data flow from program to program without ever touching a file. That one idea turns small single-purpose tools into a programming language of its own: ls | wc -l counts files, ps aux | grep nginx finds a process, sort | uniq -c builds a frequency table.

Using pipes is easy. Understanding them pays off the first time a pipeline hangs, loses an error, prints data late or kills a producer with a mysterious "Broken pipe" message. This lesson looks under the hood: how the kernel connects the processes, why they run at the same time, how exit statuses behave, what buffering does to your output, and how to make a pipe with a name.

What a pipe is

A pipe is a small in-memory buffer owned by the kernel, with a write end and a read end. When the shell runs a | b, it creates the pipe, starts both programs, connects a's fd 1 to the write end and b's fd 0 to the read end. Neither program knows about the other; a simply writes to stdout, b simply reads from stdin.

TEXT
   +-------+  stdout        kernel buffer        stdin  +-------+
   |   a   | -----------> [ ########......... ] ------> |   b   |
   +-------+  (fd 1)      (typically 64 KiB)    (fd 0)  +-------+

The rules of that buffer explain everything else:

  • If the buffer is full, the writer is paused until the reader catches up (back-pressure). A fast producer cannot flood a slow consumer.
  • If the buffer is empty, the reader waits until the writer produces data or closes its end.
  • When the writer exits (closes the write end), the reader sees end of file and finishes.
  • If the reader exits first, a writer that keeps writing receives the signal SIGPIPE and normally dies.
Bash A first pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Pipelines run concurrently

A common misconception is that a | b | c runs a to completion, then b on its results. In truth, all stages start at the same time and work in parallel on whatever data have arrived. This is why you can process a file of many gigabytes without storing the intermediate results anywhere, and why you see the first lines of output long before the input has been read completely.

You can observe it. In the following pipeline the producer writes a line, waits, writes another. The consumer prints each line as soon as it arrives. The messages of both sides are interleaved, which could not happen if the producer finished first.

Bash Stages run at the same time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

If the stages ran one after the other, all three "producer" lines would come first. Instead the consumer's lines appear between them. The consequence for scripting: a stage cannot assume the previous one has finished. A stage that must see all data first (like sort, which cannot print anything until its input ends) is a natural barrier.

Each stage also runs in its own process, which has an important side effect for variables: they cannot be changed from inside a pipeline stage. That is covered in Subshells, Groups and Environment.

Exit status of a pipeline

The exit status $? of a pipeline is, by default, the status of the last command only. Failures earlier in the pipeline are invisible:

Bash Errors in the middle of a pipeline are lost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

PIPESTATUS is a Bash array that holds the status of each command of the most recent pipeline. Copy it right away, because the very next command (even echo) replaces it. Often you only need to know whether anything failed. For that there is set -o pipefail.

set -o pipefail

With pipefail, the status of a pipeline is the status of the last (rightmost) command that failed, or 0 if every stage succeeded. Combined with set -e (exit on error) in scripts, a broken stage stops the script. This is half of the "unofficial strict mode" set -euo pipefail; Exit Codes, set -e and Error Handling explains its caveats.

Bash pipefail changes the pipeline's status

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last example is an everyday gotcha. grep exits with 1 when it finds no match. That is not an error in the usual sense, but under pipefail + set -e it terminates your script. When you accept an empty result, write grep ... || true, or handle the status explicitly.

SIGPIPE: when the reader stops early

If a consumer closes its end before the producer has finished, the next write of the producer fails and the kernel sends it SIGPIPE. By default this signal terminates the process silently. That is how head can stop an endless stream:

Bash head ends an infinite producer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Status 141 means "killed by signal 13 (SIGPIPE)", because shells report 128 + signal number. It is normal, harmless and a well-known annoyance in set -o pipefail scripts that use head. Typical remedies: ignore 141 (|| [[ $? -eq 141 ]]), use a tool that reads all its input (sed -n '1p;q' still quits early; awk 'NR==1' reads everything), or simply accept it and put || true where it is safe. See Signals and trap for more about signals.

Not all producers die quietly. A program written in a language that ignores SIGPIPE gets an EPIPE error instead and may print Broken pipe. Python does that, for example, when you pipe a long output into head.

tee: a T junction

tee copies its stdin to stdout and to one or more files. It lets you save an intermediate result while the pipeline carries on, and it is the standard way to write to a file you cannot open with > for permission reasons (... | sudo tee /etc/file; read-only here, we have no sudo).

Bash tee saves a copy while data flows on

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Buffering: why output sometimes arrives late

Programs write to stdout through a buffer in their own memory. The C library chooses the buffering mode by looking at where stdout points:

  • Terminal: line-buffered. Each completed line appears at once.
  • Pipe or file: fully buffered, typically in blocks of 4 KiB to 64 KiB. Output appears when a block is full or the program exits.

That is why tail -f app.log | grep error | awk '{print $1}' may seem to do nothing for a long time: grep (writing to a pipe) holds back its output until it has a full block. The fix is to ask the earlier stages to flush after every line: grep --line-buffered, sed -u (unbuffered), awk '{ ...; fflush() }', or run the command under stdbuf -oL.

Bash Line buffering versus block buffering

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Read the second numbers: in the first block both lines arrive after one second because grep released its block only when it exited, in the other two A arrives at second 0. Buffering only changes when data appear, never what. You notice the effect with long-running or interactive producers, such as tail -f, ping or a server log. For ordinary batch jobs it does not matter and it helps speed.

If a pipeline to a terminal seems "stuck" while the same command writing to a file works, buffering is the first suspect. Test with stdbuf -oL command or the program's own flag (--line-buffered, -u, --unbuffered).

Named pipes (FIFOs)

An ordinary pipe exists only between processes that the shell started together. A named pipe (FIFO) is a pipe that has a name in the file system, created by mkfifo. Unrelated programs can then communicate through it by opening the same path: one writes, another reads. ls -l shows it with the type letter p.

A FIFO has no data on disk; the bytes live in the kernel buffer. Opening a FIFO for reading blocks until some process opens it for writing and vice versa, so the two sides rendezvous. Start one side in the background (&) or you will wait forever:

Bash mkfifo: a pipe with a file name

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The reader loop ends only when the last writer closes the FIFO. Here one group { ...; } > channel writes both messages through a single open, so the reader sees both and then end of file. If each message were written by a separate echo "..." > channel, the reader could see end of file after the first one. For a long session keep the writing side open, for example with exec 3> channel, as in Standard Streams and Redirection. Named pipes are used for simple inter-process communication, for feeding a long-running program, and to build things like log collectors. Ready-made alternatives for scripts are process substitution (<(...), see xargs, Command Substitution and Process Substitution) and coprocesses (Advanced Patterns).

Remove FIFOs when you are done with rm; they are ordinary directory entries.

Pipelines and performance

  • Pipes are cheap. No disk is involved, and data are copied once or twice between kernel and user memory. A few extra stages cost little compared with the work they do.
  • Start fewer processes. Every stage is a new process (fork and exec). On a huge number of tiny calls (a pipe inside a loop that runs 10,000 times) the process cost dominates. Move the loop inside one awk or sed instead.
  • Filter early. Reduce the data as soon as possible, grep before sort.
  • Use all cores by design. On a multi-core machine the stages of a pipeline run in parallel for free. For one stage that is too slow, split the work with xargs -P or GNU parallel.
  • Do not cat into a filter. grep x file is faster than cat file | grep x.
  • Beware of sort. It must read all of its input before writing the first line, which breaks the streaming property; give it a hint with LC_ALL=C.
Bash A pipe inside a loop versus one process

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

All three give the same answer. With 300 lines the difference is tiny; with 3 million it is the difference between a second and many minutes. The rule of thumb: let one tool read the whole stream rather than starting a tool per line. More on this in Performance, Tricks and Pitfalls.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In a | b | c, when do the commands run?

    The shell starts every stage at once and connects them with kernel pipe buffers. Each stage works on the data that has already arrived.
  2. 2. Without pipefail, what is $? after false | true?

    By default the status of a pipeline is that of its last command. Here that is true, so $? is 0. ${PIPESTATUS[@]} would show 1 0.
  3. 3. Why does yes | head -n 1 end instead of running forever?

    When the reader closes the pipe, the next write by the producer raises SIGPIPE, which terminates it. Its exit status is 141 (128 + 13).
  4. 4. A pipeline tail -f app.log | grep ERROR | cut -c1-20 shows nothing for a long time although errors are logged. What is the most likely cause?

    When stdout is not a terminal programs use block buffering. --line-buffered (or stdbuf -oL) makes grep release each line immediately.
  5. 5. What does mkfifo jobs create?

    mkfifo creates a FIFO special file. No data are stored on disk; opening it blocks until both a reader and a writer are present.

Summary

  • A pipe is a kernel buffer connecting one process's stdout to another's stdin; a full buffer pauses the writer, an empty one pauses the reader, and closing the writer gives the reader end of file.
  • All stages of a pipeline run concurrently, so large data stream through without temporary files; sort is a natural barrier.
  • A pipeline's status is its last command's by default; ${PIPESTATUS[@]} holds every stage's status, set -o pipefail reports the rightmost failure.
  • When the reader quits early the writer gets SIGPIPE (status 141): that is how yes | head -n 1 ends, and a nuisance under pipefail.
  • tee copies the stream to files and on; |& or 2>&1 | puts stderr into the pipe too.
  • Output into a pipe is block-buffered; use --line-buffered, sed -u, fflush() or stdbuf -oL for live streams.
  • mkfifo creates a named pipe; opening blocks until both ends are present.
  • Prefer one process that reads the whole stream over one process per line.

Next: Here-Documents, Here-Strings and Input