Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 38 ยท Text Processing Pipelines

Pipelines with grep, sort, uniq, cut and AWK

Build Unix text pipelines that combine grep, cut, sort, uniq, tr, head and AWK: top-N reports, frequency tables, filters, and when to replace three tools with one AWK.

The Unix philosophy is small tools that each do one thing, joined by pipes (|) so the output of one becomes the input of the next. AWK and sed are the most flexible members of that toolbox, but they are rarely used alone. A typical real command looks like grep ... | awk ... | sort | uniq -c | sort -nr | head.

This lesson shows how the classic tools fit together, how to design a pipeline one stage at a time, and when a single AWK program is better than five commands. If any tool below is new to you, the Linux track covers it in depth.

The standard toolbox

Tool Does
grep keep (or with -v drop) lines matching a pattern
cut -d, -f1,3 pick columns by delimiter
tr 'a-z' 'A-Z' translate or delete single characters (tr -d '\r', tr -s ' ')
sort order lines (-n numeric, -r reverse, -k2,2 by field, -t, delimiter, -u unique)
uniq -c collapse adjacent duplicates and count them (so sort first)
head, tail first or last N lines
wc -l count lines
awk, sed everything else
flowchart LR
  A["data file"] --> B["grep<br/>filter rows"]
  B --> C["awk<br/>pick and compute"]
  C --> D["sort"]
  D --> E["uniq -c<br/>count"]
  E --> F["sort -nr<br/>rank"]
  F --> G["head<br/>top N"]

Read a pipeline left to right; each box only sees the previous box's text.

Build it one stage at a time

The best habit: run the first stage alone, look at the output, add the next stage, repeat. Never write a six-stage pipeline blind.

Bash Top pages from a log, step by step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Why sort before uniq

uniq only merges neighbouring identical lines. On unsorted data it counts runs, not totals.

Bash uniq needs sorted input

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Sorting by columns

sort -k selects a key: -k2,2 means "field 2 only" (without the ,2 the key runs to the end of the line). Add n for numeric and r to reverse, per key: -k3,3nr. For comma-separated data use -t,.

Bash Sorting by key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Cleaning stages: tr, cut, paste

Small tools are handy glue between the big ones:

Bash tr, cut and paste

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

When one AWK beats the pipeline

A pipeline of grep | cut | sort | uniq -c | sort -nr spawns five processes. When the logic is "filter on one field, aggregate another", one AWK is shorter, faster and easier to extend (a second total, a threshold, a header).

Bash Same answer, two ways

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

A good rule: use a pipeline when each stage is obvious and you only need counts; reach for AWK as soon as you need arithmetic, several fields at once or conditional logic.

Pipelines and exit status

In a shell pipeline the exit status is that of the last command, so a failing grep in the middle goes unnoticed. In Bash, set -o pipefail makes the pipeline fail if any stage fails. Also remember that grep exits with status 1 when it finds nothing, which is normal, not an error.

Common mistakes

  • Using uniq -c without sort.
  • Sorting numbers as text (use -n).
  • Forgetting -t for sort on CSV, so spaces inside values break the keys.
  • Quoting: wrap the AWK/sed program in single quotes so the shell does not expand $1.
  • Writing cat file | grep x: grep x file does the same without an extra process.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does uniq -c usually follow sort?

    uniq compares each line with the previous one, so identical lines must be next to each other, which sorting guarantees.
  2. 2. What does sort -t, -k3,3nr do?

    -t, sets the separator, -k3,3 limits the key to field 3, n means numeric and r reverses.
  3. 3. In a pipeline, which command's exit status is reported by default?

    Without pipefail the shell reports only the last command's status.
  4. 4. When is a single AWK program a better choice than grep | cut | sort | uniq?

    AWK combines filtering, field handling and arithmetic in one process, so it is shorter and easier to extend.

Summary

  • Connect small tools with pipes and build the pipeline one stage at a time.
  • sort | uniq -c | sort -nr | head is the classic frequency ranking.
  • sort -k, -n, -r, -t control ordering; sort before uniq.
  • tr, cut and paste are useful glue; AWK replaces several stages when arithmetic is involved.
  • Pipelines report the last stage's status unless you set pipefail.

Next: Working with Several Files and Joins in AWK