Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 38 ยท Getting Started

How Text Streams Work: Lines, Fields and Pipes

Understand the model behind sed and AWK: standard input and output, lines and records, fields, exit status, and how pipes connect small tools into pipelines.

Before you write any sed or AWK, you need a clear picture of what these tools see. They do not see a "document" with pages and paragraphs. They see a stream of characters, cut into lines, and (for AWK) lines cut into fields. Once that model is in your head, most commands become easy to predict.

This lesson covers standard input and output, the idea of a record, how to feed data to a tool in several ways, and how pipes join tools together.

Standard input, output and error

Every Unix program starts with three open channels:

Channel Number Default Used by sed/AWK for
stdin 0 the keyboard or a pipe input when you give no file name
stdout 1 the screen the edited text
stderr 2 the screen error messages

sed and AWK read from stdin if you give no file, and write results to stdout. They never change their input by themselves. That is why you can safely experiment: sed 's/a/b/' notes.txt only prints a changed copy.

Bash Four ways to give input

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

All four forms give the tool the same thing: a stream of lines. The only difference is who opens the file. With several file names, sed and AWK read them one after another as if they were one long stream.

Lines are records

A record is a chunk of input that the tool processes in one go. By default a record is a line, ended by a newline character (\n). sed calls the current record the pattern space; AWK stores it in $0 and counts records in NR.

TEXT
  input bytes:  a b c \n d e \n f g h \n
                \_____/  \___/   \_____/
                 line 1  line 2   line 3     <- records

If the last line has no final newline, sed and AWK still process it. Be aware when files come from Windows: lines may end with \r\n, and the invisible \r becomes part of the last field. You will meet cleanup recipes in Real-world Tasks.

Fields

AWK goes one step further: it splits each record into fields. By default, fields are separated by runs of spaces and tabs, and leading blanks are ignored.

Bash Lines and fields in AWK

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

NR is the record number, NF is the number of fields, and $NF is the last field. Notice that ada lovelace 1815 has several spaces, but AWK still finds three fields. sed has no built-in idea of fields; it works with characters and regular expressions, which is why sed is better at text surgery and AWK at tables.

Pipes: building pipelines

The pipe | connects the stdout of one program to the stdin of the next. Both programs run at the same time, passing data in small pieces, so a pipeline can handle files much bigger than memory.

flowchart LR
    F["file or generator"] -->|"stdout to stdin"| S["sed"]
    S -->|"pipe"| A["awk"]
    A -->|"pipe"| O["sort"]
    O --> R["terminal"]

Each program reads lines and writes lines.

Bash A first pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

grep filters, awk totals per region using an array, and sort puts the output in a stable order (AWK's for (r in sum) order is not defined, so sorting afterwards is a good habit).

Exit status

Every command ends with a number: 0 means success, anything else means trouble. AWK exits with 0 normally; sed exits 0 too, unless it hits an error such as a bad script (status 1) or a missing file (status 2). sed -n '/x/p' does not fail when nothing matches, which surprises people who expect it to behave like grep. In a script, check $? or use set -e carefully.

Bash Exit codes of sed and grep

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Quoting the program

Both tools take their program as a command-line argument, and the shell gets a chance to mangle it first. Use single quotes so the shell leaves $1, * and ! alone:

TEXT
awk '{ print $1 }' file     # right: AWK sees $1
awk "{ print $1 }" file     # wrong: the SHELL replaces $1 with its own argument

Need a single quote inside the program? Stop the quoting, add an escaped quote, and resume: 'it'\''s'. For longer programs, put them in a file (-f script) and avoid quoting issues entirely. See sed script files.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does sed 's/a/b/' file.txt do to file.txt?

    sed writes to stdout by default. Only -i (in-place) edits the file, and that comes later in the course.
  2. 2. What do NR and NF hold in AWK?

    NR counts the records read so far (usually line numbers) and NF is the number of fields in the current record.
  3. 3. Why quote an AWK program with single quotes in the shell?

    Inside single quotes the shell passes everything literally, so $1 reaches AWK instead of being replaced by a shell variable.
  4. 4. What exit status does sed -n '/zzz/p' return when nothing matches?

    Unlike grep, sed finishes successfully even when its pattern matches no line.

Summary

  • sed and AWK read stdin or the named files and write to stdout, leaving inputs untouched.
  • A record is normally a line; AWK splits it into fields ($1, $2, NF, $NF).
  • Pipes connect small programs into pipelines that stream data.
  • Use single quotes around programs and check exit statuses when scripting.

Next: Regular Expressions for sed and AWK