Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 95 ยท Pipes and Redirection

Standard Streams and Redirection

Understand stdin, stdout, stderr and file descriptors in Bash: redirect with >, >>, <, 2>, 2>&1 and &>, the order of redirections, noclobber and exec.

Every program you start on Linux is born with three open channels for data. One carries what the program reads (standard input), one carries its normal results (standard output) and one carries its error messages (standard error). Normally all three are connected to your terminal: you type, results and errors appear on screen. Redirection means reconnecting these channels to files, to other programs or to nowhere. It is the foundation of shell scripting: it lets you save output, feed a file to a program, silence noise, and separate results from errors.

The idea is simple, but the syntax has traps, especially the order in which redirections are written. This lesson builds the model slowly: first what the three streams are, then every operator, then the order rules, then the scripting techniques (exec, loops, noclobber, file descriptors above 2).

The three standard streams

The kernel identifies every open file of a process with a small number, a file descriptor (fd). The first three are fixed by convention:

fd Name Default Used for
0 stdin keyboard (terminal) data the program reads
1 stdout screen normal results
2 stderr screen error messages and diagnostics
TEXT
              +-----------+
 keyboard --> | 0 stdin   |
              |  program  | 1 stdout --> screen
              |           | 2 stderr --> screen
              +-----------+

Why two output streams? Because results and errors are different kinds of information. If you run ls /nonexistent > list.txt, the error message should still reach your eyes, not vanish into the file, and a pipe should pass only results to the next program. Programs write results to fd 1 and complaints to fd 2.

In our runner stdout and stderr are shown as two separate blocks. On a terminal they interleave. We add 2>&1 to some examples to merge them in the correct order; you will learn exactly what that means below.

Bash Output and error are different streams

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In the second command, ls found exists.txt (its name went to stdout, into out.txt) and failed on missing.txt (the complaint went to stderr, which was still the screen). That separation is the whole point.

Redirecting output: >, >> and 2>

  • command > file sends stdout to the file, creating or truncating it. The shell opens the file before the command starts, so even a command that prints nothing empties the file.
  • command >> file appends instead of truncating.
  • command 2> file redirects stderr (fd 2). A number in front of the operator selects the descriptor; plain > means 1>.
  • command > file 2> errors handles both separately.
Bash Truncate, append and capture errors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

> file destroys the old content immediately. A classic accident is sort data.txt > data.txt: the shell empties data.txt first, so sort reads an empty file and the data are gone. Write to a temporary file and move it back (sort data.txt > tmp && mv tmp data.txt), or use sort -o data.txt data.txt, which sort handles safely.

Redirecting input: <

command < file connects the file to stdin (fd 0). The program cannot tell it is reading a file and not a keyboard. Many programs behave differently when given a file name argument versus stdin: wc -l file prints N file, but wc -l < file prints just N, because wc never learned the name.

Bash Reading from a file with <

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The shell performs < and > before the command runs and then removes them from the argument list. The program only ever sees its arguments, never the redirections. That is also why redirections can appear anywhere on the command line, even at the start: > out.txt echo hi works.

Merging streams: 2>&1 and &>

2>&1 means "make fd 2 a copy of whatever fd 1 points to right now". After it, errors travel the same road as results, so they land in the same file or pipe. Remember the shape: N>&M duplicates descriptor M onto N. The & marks that M is a descriptor, not a file name. Without it, 2>1 would write errors to a file named 1.

Bash also has the shorthand &> file, equal to > file 2>&1. It is a Bash extension: in a POSIX sh script it means "run in the background" and then redirect, with surprising results. For portable scripts write the long form.

Bash Combining stdout and stderr

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In the pipe examples notice the difference. With 2>&1 | the error text is upper-cased as well. Without it, the error message bypasses tr and arrives unchanged in the stderr block. The pipe operator itself is covered in Pipes and Pipelines; there is also the shorthand |& for 2>&1 |.

The order of redirections matters

Redirections are processed left to right, and 2>&1 copies the current target of fd 1. This explains the most famous trap in Bash:

TEXT
command > file 2>&1     stdout -> file, then stderr -> where stdout goes now (file)
command 2>&1 > file     stderr -> where stdout is now (screen), THEN stdout -> file

In the second line, 2>&1 happens first, when fd 1 still points to the terminal. Stderr gets the terminal. Only afterwards is stdout moved to the file. The errors stay on screen.

Bash 2>&1 order: > file 2>&1 versus 2>&1 > file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In block B the error message appears in the stderr block, and B.txt has only the result. A handy trick follows from this order: command 2>&1 >/dev/null | next sends only the errors through the pipe, because stderr is copied to the pipe first and stdout is thrown away afterwards.

Bash Swap streams: send only errors into a pipe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

/dev/null: the black hole

/dev/null is a special file that accepts every write and discards it; reading from it gives an immediate end-of-file. Use it to silence a stream:

  • command > /dev/null hides normal output.
  • command 2> /dev/null hides errors (but also hides real problems: do it only when you know what you are discarding).
  • command > /dev/null 2>&1 hides everything; you then rely on the exit status ($?) or if command; then.
  • command < /dev/null gives the program an empty stdin, so that it never waits for the keyboard.
Bash Silencing output and using the exit status

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Redirection in loops, groups and functions

A redirection attached to a compound command applies to everything inside it. That is more efficient and tidier than redirecting each line: the file is opened once.

Bash Redirecting a whole loop or group

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The while read ... done < file form is the right way to read a file line by line; Loops shows the details such as IFS= and -r. Notice that redirecting a loop opens the file once, whereas echo ... >> file inside the loop would open and close it on every iteration.

Protecting files: noclobber

set -o noclobber (or set -C) makes > refuse to overwrite an existing regular file, a safety net against > data.txt accidents. To overwrite on purpose, use the override >|. >> is unaffected. Most people enable this in interactive shells and in scripts that handle valuable files.

Bash noclobber protects existing files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The refused redirection makes the shell itself print main.sh: line 6: important.txt: cannot overwrite existing file. That message is written before the redirections of that command are in place, which is why it shows up in the stderr block even though the command says 2>&1.

Here documents and strings in one line

Two more input forms exist: <<EOF (a here-document: the lines up to EOF become stdin) and <<< "text" (a here-string). They deserve their own lesson: Here-Documents, Here-Strings and Input.

exec: redirecting the whole script

exec with no command does not start anything; it changes the redirections of the current shell for the rest of the script. This is how scripts log everything without decorating every line:

Terminal
exec > build.log 2>&1      # from now on, all output and errors go to build.log

You can also open your own descriptors (3 and above). exec 3> file opens fd 3 for writing, exec 4< file opens fd 4 for reading, >&3 writes to it and exec 3>&- closes it. A common use is saving the original stdout so you can restore it later.

Bash exec and custom file descriptors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Once exec > captured.txt has run, even the shell's own echo goes into the file. Restoring needs the saved copy on fd 5. This pattern is how test frameworks capture the output of a code section. A trick for logging only part of a script is to wrap that part in { ...; } > file instead of using exec.

Reading and writing the same descriptor

<> opens a file for both reading and writing, and /dev/stdin, /dev/stdout and /dev/stderr are names for fds 0, 1 and 2 on Linux. Some programs, like tee, accept /dev/stderr as a "file" so that a copy of the data is sent to the error stream (/dev/fd/N also names descriptor N). Availability varies in restricted environments such as our sandbox, so we do not run that here.

CONSOLE
$ echo "debug info" | tee /dev/stderr | wc -c      # read-only: needs a normal terminal
debug info
11

Redirection and error messages: a common set of idioms

TEXT
cmd > out.txt             save results, errors still visible
cmd >> out.txt            append results
cmd 2> err.txt            save errors only
cmd > out.txt 2>&1        everything in one file (portable)
cmd &> out.txt            everything in one file (Bash only)
cmd > /dev/null 2>&1      run silently, test $?
cmd 2>&1 | other          pipe results AND errors
cmd 2>&1 >/dev/null | x   pipe errors ONLY
cmd < in.txt              feed a file
cmd >&2                   (inside a script) print a message to stderr

That last one matters for scripts: diagnostic messages belong on stderr, so that ./script > result.txt still shows them to the user and does not pollute the data. Write echo "error: file missing" >&2, or better printf 'error: %s\n' "$msg" >&2.

Bash Messages to stderr from a script function (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last command exits with status 1, and the runner notes that under the output. Because die printed to stderr, the message error: this stops the script appears in the error block, and never reached is never printed. Exit statuses are the topic of Exit Codes, set -e and Error Handling.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which file descriptor number is standard error?

    fd 0 is stdin, fd 1 is stdout and fd 2 is stderr. Descriptors from 3 upwards are free for your own files.
  2. 2. What is the effect of ls a nope 2>&1 > out.txt?

    Redirections are processed left to right. 2>&1 copies the terminal as stderr's target first, then > moves only stdout to the file. To capture both, write > out.txt 2>&1.
  3. 3. Why is sort data.txt > data.txt dangerous?

    The shell opens and truncates the output file before starting the command. Use a temporary file, or sort -o data.txt data.txt.
  4. 4. What does command > /dev/null 2>&1 do?

    /dev/null throws away everything written to it. Sending stdout there and then pointing stderr at the same place silences the command completely; you can still check $?.
  5. 5. What does exec 3> notes.txt do in a script?

    exec without a command only changes redirections of the current shell. Here it opens fd 3, which later commands can use with >&3.

Summary

  • Each process has stdin (fd 0), stdout (fd 1) and stderr (fd 2). Results go to stdout, messages and errors to stderr.
  • > truncates, >> appends, < reads from a file, 2> redirects errors, 2>&1 copies stdout's current target onto stderr, and &> is the Bash shorthand for both.
  • Redirections are processed left to right: > file 2>&1 captures both, 2>&1 > file does not.
  • /dev/null discards output; combine with the exit status to run commands silently.
  • A redirection on a loop, group or function applies to everything inside; set -o noclobber protects existing files, >| overrides it.
  • exec changes the shell's own redirections and opens extra descriptors (exec 3> file, >&3, exec 3>&-).
  • In scripts, send diagnostics to stderr with >&2 so results stay clean.

Next: Pipes and Pipelines