Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 95 ยท Working with Text

Building Text Pipelines

Learn to compose grep, sed, awk, sort and uniq into reliable text pipelines: log analysis, CSV summaries, process substitution and debugging with tee.

You now know a toolbox: grep searches, sed edits, awk computes, sort, uniq and wc order and count. Each does one thing. The real power of Linux is that they all speak the same language, lines of text on standard input and output, so you can connect them with the pipe character and build a custom data tool in one line.

This lesson is about the craft of composing. How do you decide which tool does which step? How do you build a long pipeline without getting lost? How do you debug one that prints nonsense? And when is a pipeline the wrong answer? We will work with realistic data: a web server log, a CSV file with a header and some dirty rows, and a text with words to count.

The mental model: a conveyor belt

A pipeline is a conveyor belt of text. Each station takes lines in, does one transformation and passes lines on. You design it from the left: start with the raw data, and at every step ask "what shape should the text have now?"

TEXT
raw log  ->  keep relevant lines  ->  pick the column  ->  group  ->  count  ->  rank  ->  top 5
 cat/file     grep / awk pattern      cut / awk '{print}'   sort      uniq -c    sort -rn   head

The most reliable way to build one is incrementally. Write the first stage, run it, look at the output. Add the next stage, run again. Never write six stages blind and then wonder why the result is empty. Interactive shells make this easy: press the up arrow, append | next-stage, run.

Stage one: the data

Everything in this lesson uses a small log in the "common" shape of web server logs. Each TryIt builds its own copy, because every example starts with a fresh directory.

Bash Build the pipeline step by step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Each line adds exactly one stage to the previous one. When you read a pipeline, read it left to right as a sentence: "take the lines with 200, take the path column, sort, count, rank, show the first."

Choosing the right tool for each stage

A quick guide that experienced users follow without thinking:

Need Tool Why
Keep or drop lines matching a pattern grep fastest, simplest
Replace text inside lines sed 's/old/new/' substitution is its specialty
Pick columns, compute, group, format awk fields, variables, arithmetic
Pick a character range or one delimiter column cut tiny and fast
Reorder, group, count sort, uniq -c built for it
Change or delete single characters tr character-level
First or last N lines head, tail streams stop early

A tool can often do the job of the previous two (awk '/200/ {print $6}' replaces grep | awk). Merging stages is an optimization you can do later, once the pipeline works. In the beginning, clarity beats cleverness: separate stages are easy to test one by one.

Log analysis recipes

Here are four questions you would ask of any log, with the one-line answers.

Bash Log analysis: clients, errors, traffic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice $(NF-1): NF is the number of fields, so $NF is the last and $(NF-1) the one before it. Anchoring on the end of the line is more robust than counting from the start when the early fields can vary in number. The awk '$(NF-1) >= 400' form is a pattern: awk prints (or acts on) only the lines for which the condition is true. All of this is explained in depth in awk.

Cleaning and summarizing CSV data

CSV files from the real world come with a header, blank lines, inconsistent case and stray spaces. A cleaning pipeline normalizes them before analysis. Always handle the header explicitly: tail -n +2 drops it, or awk 'NR > 1'.

Bash Clean a messy CSV and summarize it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The cleaning line has three stages: grep -v '^$' removes empty lines, sed removes spaces around commas and at the end of lines, and awk lower-cases column 2 (setting OFS=, so that awk joins fields back with commas). The summary uses an awk associative array, total[$2] += $4: a table indexed by region. The for (r in total) loop visits keys in an unspecified order, so we pipe into sort to make the result stable. Never depend on the order of an awk loop.

These recipes assume simple CSV. If a field can contain a comma inside quotes ("Smith, John") or a newline, splitting on , is wrong. For anything but generated, well-behaved data, use a real CSV tool or a scripting language. The shell is excellent at line-oriented text and weak at structured formats.

Word frequencies and text statistics

Counting words is the "hello world" of text processing and shows how tr turns a text into one item per line.

Bash Word frequencies with a normalizing pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

tr -cs 'a-z' '\n' is the key stage: -c complements the set (everything that is not a lowercase letter), -s squeezes repeats, and every such run becomes a single newline. Punctuation and spaces disappear and each word is on its own line. The grep -c . counts non-empty lines (the first line of the translated output can be empty).

Process substitution: pipelines with several inputs

A pipe has exactly one input. What if a command needs two streams, like diff or comm or paste? Process substitution <( command ) runs the command and gives its output a temporary file name, so any program that expects a file can read it. You can even use several in one command.

Bash Process substitution with diff, comm and paste

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Process substitution is a Bash feature (also in zsh and ksh), not in POSIX sh or dash. It is the cleanest way to avoid temporary files. The >( ) form is the inverse: it presents a command as a file you can write to. Because those commands run asynchronously, we sleep briefly in the demo; in real scripts prefer plain pipes when you can. We cover <( ) further in xargs, Command Substitution and Process Substitution.

Debugging a pipeline with tee

When a long pipeline gives a wrong answer you need to see the data between the stages. Two techniques:

  1. Run it in pieces: execute the first N stages and look at the output (what we did above).
  2. Insert tee somewhere: tee file copies its input to a file and also passes it on, so the pipeline runs unchanged while you capture a snapshot.

On a normal terminal, tee /dev/stderr shows the data on the screen without altering the rest (that device is missing in our sandbox, so the example below saves a file instead). head and wc -l at various points are the other usual instruments: check the line count after each stage and see where lines disappear.

Bash tee as a probe between stages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

A common cause of "my pipeline prints nothing" is a stage that filters everything out: a grep pattern that does not match because of a hidden carriage return, a different case, or a field that is not where you thought. The probe shows you immediately.

Streaming and performance

Pipelines are fast because the stages run at the same time: the kernel connects them with a small buffer, and each program works on lines as they arrive. A 50 GB log can flow through grep | awk | sort without ever being held in memory, except in the sort stage, which needs all data to order it (and spills to temporary files when memory is short).

A few rules of thumb:

  • Filter early. Put grep as far left as possible so later stages see fewer lines.
  • Avoid cat file | at the front. grep pattern file or < file command skips a process.
  • LC_ALL=C makes sort, grep and awk much faster on big ASCII files.
  • head ends early. The upstream commands are stopped with SIGPIPE when head has enough (Pipes and Pipelines).
  • Merge stages that walk the same data. One awk that filters, extracts and sums is faster than grep | cut | awk.
  • Do not parse ls. Use globs or find for file lists.
Bash Merging stages: same answer, fewer processes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Do not obsess about speed for small inputs: for a few thousand lines any version is instant. Start with the readable pipeline and optimize only when the data are large.

When a pipeline is the wrong tool

Pipelines shine for line-oriented, one-shot analysis. They are the wrong tool when:

  • the data are structured (JSON, XML, quoted CSV): use jq or a parser in a real language;
  • you need state across many records with complex logic: an awk program or a Python script is easier to read;
  • the one-liner has grown beyond about 5 stages and nobody can understand it: write a script with named variables and comments (see Writing Your First Script).

A 3-line awk program in a file often beats a 200-character one-liner. Readability is a feature.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the best way to build a long pipeline?

    Building incrementally shows you the shape of the data at each step, so the stage that goes wrong is obvious.
  2. 2. Why does a pipeline sort | uniq -c | sort -rn need two sorts?

    uniq only merges neighbouring lines, so the first sort puts equal values side by side. After counting, the second sort ranks groups by count.
  3. 3. What does diff <(sort a.txt) <(sort b.txt) demonstrate?

    <( ... ) runs the command and gives its output a file name (usually /dev/fd/N), so diff can read two streams.
  4. 4. Which tool helps you see the data between two stages of a pipeline without changing the result?

    tee copies its input to a file (or /dev/stderr) and also passes it on, so it works as a probe.
  5. 5. In awk, why use $NF or $(NF-1) for log fields near the end of a line?

    NF is the number of fields on the current line, so $NF is always the last field, whatever the line length.

Summary

  • A pipeline is a conveyor belt of lines: filter, extract, group, count, rank. Build it one stage at a time and look at the output at each step.
  • Pick the simplest tool per stage: grep filters, sed substitutes, cut/awk pick columns, sort/uniq group, tr handles characters.
  • $NF and $(NF-1) address fields from the end of the line; awk arrays group and sum, and their key order is undefined, so sort the result.
  • Normalize data first (case, spaces, blank lines, header) and be careful with real CSV, which the shell cannot parse safely.
  • <( ) process substitution gives commands a file name for a stream; tee is the probe for debugging.
  • Filter early, merge stages when needed, set LC_ALL=C for speed, and move to a script when the one-liner stops being readable.

Next: Standard Streams and Redirection