Skip to content
Tutorials Center
My learning

Topics

Lesson 63 of 95 ยท Advanced Bash

Text Processing Recipes

A cookbook of shell text recipes: dedupe, set operations, joins, column sums, bulk rename, key=value parsing, JSON with grep and sed, CSV pitfalls and templates.

Once you know grep, sed, awk, sort and a few friends, most day-to-day text jobs are small variations on a handful of patterns. This lesson is a cookbook of those patterns. Each recipe states the problem, shows the shortest correct solution, and says where it breaks, because every one-liner has an edge where it stops being safe.

The tools themselves are taught in grep, sed, awk and sort, uniq, cut, tr and wc, and Building Text Pipelines shows how to chain them. Here we put them to work. Every example creates its own sample data, so you can run and modify it freely.

Removing duplicates

There are three different questions hiding behind "dedupe", and each has its own answer.

  • Sorted output is fine: sort -u.
  • Keep the first occurrence, preserve the original order: awk '!seen[$0]++'.
  • Show only the lines that appear more than once: sort | uniq -d.

The awk version deserves a closer look. seen[$0] is an array element indexed by the whole line; the first time a line occurs it is 0 (false), !0 is true so awk prints the line, and then ++ makes it 1. Every later copy finds a non-zero value and is skipped.

Bash Three ways to remove duplicates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

To dedupe on one field instead of the whole line, index the array by that field: awk -F, '!seen[$1]++' data.csv keeps the first row for each value of column 1.

Comparing and combining lists

comm compares two sorted files and prints three columns: lines only in the first, only in the second, and in both. Suppressing columns gives you the classic set operations.

Goal Command
In A and B (intersection) comm -12 a b
Only in A (difference) comm -23 a b
Only in B comm -13 a b
In A or B (union) sort -u a b

join goes further: it combines two sorted files on a common key, like a database join. Both inputs must be sorted on the join field with the same locale, or join misses matches (the sandbox sets LC_ALL=C, which is a good habit in scripts).

Bash Set operations with comm and a join on a key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Fields and columns

For fixed or whitespace-separated columns, awk is the most flexible extractor; cut is faster and simpler when the delimiter is a single character.

Summing a column is the recipe people reach for most often. With a header row, skip it using NR > 1. To group (a "pivot"), use an associative array keyed by the group column.

Bash Column sums and group-by with awk

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The for (r in rev) loop visits keys in an unspecified order, so the example pipes the result to sort. Never rely on awk's array order in output you compare or test.

awk -F, splits on every comma. It cannot tell a comma inside a quoted field from a separator. The CSV section below shows what goes wrong and what to use instead.

Bulk renaming files

The shell can rename files with a loop and parameter expansion, with no external tool. The structure is always the same: loop over a glob, build the new name, check for collisions, then mv -- "$old" "$new". Do a dry run first by putting echo in front of mv or by printing the plan.

Bash Rename with a loop, collision-safe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

${old,,} lower-cases the value (Bash 4+), ${var//a/b} replaces every a, and ${f%.jpg} strips a suffix. The -e test is the collision check: mv silently overwrites an existing file unless you add -n (no clobber). Debian ships a rename command, but its syntax differs between distributions (a Perl script on Debian and Ubuntu, a different util-linux tool on Fedora), so scripts that must travel use the loop.

Parsing key=value files

Configuration-style files (NAME=value) are everywhere. To read one key, grep plus cut is enough. To read all keys into an associative array, loop with IFS='=' read.

Bash Reading key=value settings safely

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

IFS='=' read -r key value splits at the first = only, because the last variable receives the rest of the line, so values may themselves contain =. Do not source such files unless you fully trust them: sourcing executes whatever the file contains.

JSON with grep and sed, and why to use jq

For flat, predictable JSON you can sometimes pull a value out with sed. This works only while the JSON is formatted exactly the way you expect:

Bash A fragile JSON one-liner (and where it breaks)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first two commands work. The multi-line file also works here because sed reads line by line and "name" happens to sit on one line; but if the key and value were on different lines, or if there were two "name" keys at different nesting levels, the regex would silently pick the wrong one. The last example prints Ada \ (stopping at the escaped quote), which is wrong. Regular expressions cannot parse nested or escaped structures reliably.

The right tool is jq, a JSON processor. It is not installed in our sandbox, so these are read-only examples:

Terminal
# needs jq; not runnable here
jq -r '.name' one.json                       # Ada
jq -r '.tags[]' one.json                     # math  code (one per line)
jq '[.[] | select(.age > 30)]' people.json   # filter an array

Rule of thumb: a throwaway grep -o for a quick look at a log line is fine. For anything a script depends on, use a real parser.

CSV pitfalls

CSV looks like "text separated by commas", and that is exactly the trap. The format allows commas and even newlines inside quoted fields, and that breaks any tool that just splits on ,:

Bash Why cut and awk break on quoted CSV

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Your options, in order of preference:

  1. Use a CSV-aware tool: csvkit, Miller (mlr), or a language with a CSV library. None is installed in our sandbox.
  2. If the data never contains quoted commas (you generated it, or you control the producer), awk -F, and cut are fine. Say so in a comment.
  3. For one-off jobs, a short Python script is clearer than a regex monster. This is a labelled stand-in, since Python is not part of the Linux toolset we teach, but it shows what correct parsing needs:
Bash A CSV-aware stand-in using python3

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Other CSV traps: a byte-order mark at the start of the file (from Excel) glues itself to the first header name; Windows line endings leave a trailing \r on the last column (strip with tr -d '\r'); and numbers with locale-specific decimal commas.

Templating

Sometimes you need to produce a file from a template: a config with the host name filled in, an email with the user's name. There are three levels of tooling.

A here-doc generator is the simplest. An unquoted delimiter lets the shell expand variables and commands in the body (see Here-Documents).

envsubst (from the gettext package) replaces $VAR in a template file with the environment. It is not installed here, so that is read-only: NAME=Ada envsubst < template.txt > out.txt.

A hand-written substitution function needs no extra package and lets you control which placeholders exist:

Bash Here-doc generator and a placeholder function

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The substitution ${text//"{{key}}"/value} quotes the pattern so that the braces are literal. Because this is plain string replacement, a value containing & or \ is safe (unlike in sed's replacement part, where & means "the whole match").

Choosing the tool

A quick decision guide for text recipes:

  • Match or filter lines: grep.
  • Replace text inside lines: sed.
  • Work with columns, numbers or groups: awk.
  • Order, count or dedupe: sort, uniq.
  • Fixed-width slicing or single-character delimiters: cut.
  • Anything nested or quoted (JSON, XML, real CSV): a parser, not a regular expression.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which command removes duplicate lines but keeps the first occurrence of each, in the original order?

    The awk array remembers lines already printed. sort -u removes duplicates but sorts, uniq only collapses adjacent duplicates, and uniq -d prints only the duplicated lines.
  2. 2. What does comm -12 a b print (both files sorted)?

    comm prints three columns: only-in-first, only-in-second, in-both. -12 suppresses the first two, leaving the lines common to both files.
  3. 3. Why does IFS='=' read -r key value work for lines like greeting=hello = world?

    When there are more fields than variables, the last variable keeps the remainder unsplit. So key is greeting and value is hello = world.
  4. 4. Why is awk -F, risky for real-world CSV files?

    CSV allows commas (and newlines) inside quoted fields. Splitting on every comma produces extra fields. Use a CSV-aware tool unless you know the data has no quoted commas.
  5. 5. What is the main reason to prefer jq over grep and sed for extracting values from JSON?

    Regular expressions cannot reliably handle nested objects, escaped quotes or different formatting. A real parser can, which is why scripts that depend on JSON should use jq.

Summary

  • Dedupe with sort -u (sorted), awk '!seen[$0]++' (first occurrence, order kept) or uniq -d (only duplicates).
  • Use comm for set operations on sorted lists and join for key-based merges; keep locale and sorting consistent.
  • awk handles column sums, averages and group-by with associative arrays; sort the output because array order is unspecified.
  • Bulk renames are loops with parameter expansion, a collision check and mv --; dry-run first.
  • Parse key=value with IFS='=' read -r; never source untrusted files.
  • Regex one-liners on JSON and CSV are fragile; use jq or a CSV-aware tool when the data matters.
  • Template with a here-doc, envsubst or a small replacement function.

Next: Performance, Tricks and Pitfalls