Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 38 ยท Text Processing Pipelines

Combining sed and AWK in Scripts

Learn when to use sed and when to use AWK, how to chain them in pipelines, share shell variables safely, write executable scripts and avoid quoting pitfalls.

sed and AWK overlap, but they are good at different things. sed is a stream editor: it excels at "find this text, change it, delete those lines, insert something here". AWK is a data language: it excels at "split into columns, calculate, remember, summarise". Most strong command lines use each for what it does best, stitched together with pipes.

This lesson gives you a decision rule, shows combined pipelines and then covers the practical glue: shell variables, quoting, scripts and portability between GNU and BSD versions.

Choosing the right tool

You want to... Reach for
Replace text, delete or insert lines sed
Print a line range (sed -n '10,20p') sed
Edit a file in place sed -i
Pick or reorder columns awk
Calculate (sum, average, max) awk
Count or group by a key awk
Conditional logic on field values awk
Reformat with padding and alignment awk (printf)
flowchart LR
  A["messy text"] -->|"sed: strip junk,<br/>normalise"| B["clean lines"]
  B -->|"awk: columns,<br/>maths, groups"| C["numbers and tables"]
  C -->|"sed: final touches"| D["report"]

A common shape: sed cleans, AWK computes, sed polishes.

sed first, then AWK

Raw input often has decoration that gets in the way of fields: currency signs, thousands separators, quotes, Windows line endings, comments. Let sed remove it so AWK sees plain numbers.

Bash Clean with sed, total with AWK

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The sed program deletes comment lines and blank (or whitespace-only) lines, then strips dollar signs and commas. AWK splits on the colon and adds the second field (AWK ignores the leading spaces when converting " 1200.00" to a number).

AWK first, then sed

The opposite order also works: compute with AWK, then use sed to tidy the text of the result.

Bash Compute with AWK, decorate with sed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last line uses a loop (:a label and ta "branch if a substitution happened") that you will meet properly in the sed scripting chapter. Here it repeatedly inserts a comma before the last three digits.

Shell variables and quoting

The program text of both tools is usually in single quotes, so the shell leaves $1 and \( alone. That also means a shell variable is not expanded inside. Three safe ways to bring a value in:

  1. AWK: awk -v n="$n" '$2 > n' (value passed as data).
  2. AWK: ENVIRON for arbitrary text.
  3. sed: close the single quotes, insert the variable in double quotes, reopen: sed 's/foo/'"$x"'/'. This is only safe if $x has no /, & or backslash, because sed will treat it as part of the replacement syntax.
Bash Passing shell variables

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

For any value you do not control, the escape step (sed 's/[\/&]/\\&/g') or a different delimiter protects you from / and & surprises.

Writing a script file

Once a pipeline has more than two stages, put it in a file with a shebang line, a comment for each stage and set -eu so errors do not pass silently.

Bash A small report script

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

sed and AWK in one script

AWK can also run sed-like edits itself (sub, gsub), and sed can do tiny bits of arithmetic with tricks, but that is not what it is for. A handy compromise: let AWK generate sed commands (or shell commands) and pipe them to the interpreter. For example, to rename files:

Terminal
ls *.txt | awk '{ new = $0; sub(/\.txt$/, ".md", new); print "mv \"" $0 "\" \"" new "\"" }'

Look at the generated commands first; when they look right, append | sh. Generating the commands makes a dry run free.

GNU and BSD differences

Scripts you share must survive other machines:

  • sed -i needs a suffix on macOS/BSD (sed -i ''), but GNU sed takes none attached (-i or -i.bak). Using -i.bak works on both.
  • \+, \?, \| in basic regexes and \n in replacements are GNU extensions; sed -E (extended regexes) is understood by modern GNU and BSD.
  • Different AWKs (gawk, mawk, nawk) differ in intervals {n,m}, length(array), strftime, and asort. Stick to POSIX features for portable scripts.

Common mistakes

  • Using double quotes around the AWK program, so the shell expands $1.
  • Interpolating variables into sed expressions without escaping / and &.
  • Using sed -i without a backup suffix on a file you cannot recreate.
  • Doing arithmetic in sed or text surgery with many substr calls in AWK: pick the tool whose strength fits.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which task is the best fit for AWK rather than sed?

    Grouping and arithmetic need variables and arrays, which are AWK's strength. The others are classic sed jobs.
  2. 2. Why use awk -v n="$n" instead of pasting $n into the AWK program text?

    -v passes data separately from the code, which avoids quoting bugs and injection.
  3. 3. What is a portable way to edit in place with a backup?

    With an attached suffix, -i.bak works in both GNU and BSD sed.
  4. 4. Why choose a different delimiter, like s|BASE|$path|, when the replacement is a path?

    sed lets you use any character as the s delimiter; picking one absent from the data avoids escaping.

Summary

  • Use sed for editing text and lines, AWK for fields, arithmetic and grouping.
  • Common shape: sed cleans, AWK computes, sed polishes.
  • Bring shell values in with -v or ENVIRON; escape / and & for sed or change the delimiter.
  • Put multi-stage pipelines in scripts with a shebang and set -eu.
  • Watch GNU, BSD and AWK-version differences when sharing scripts.

Next: Multi-line Records, Paragraph Mode and CSV in AWK