Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 38 ยท One-liners

Text Surgery One-liners: sed and AWK Together

One-liners for everyday text surgery: insert, append and replace lines, comment out settings, strip HTML tags, extract emails and URLs, convert TSV to CSV, and edit many files.

The previous two lessons collected one-liners for each tool on its own. This one is about operations: the small jobs you do again and again to files, such as "add a line after the match", "comment out this setting", "pull all the URLs out of this page", "convert the separators". Each recipe picks whichever of sed, AWK or a combination reads best.

Everything below creates its own input, so you can run, modify and re-run it. The -i (in-place) examples work on a copy inside the editor, never on a file you care about.

Inserting, appending and replacing whole lines

GNU sed has three commands for whole-line edits: i\ (insert before), a\ (append after) and c\ (change, replacing the line). On one line you can write sed '/re/a text'.

Bash i, a and c

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The one-line form of a, i and c is a GNU convenience; the portable form puts a backslash and a newline after the command. With several -e options, each -e is one script line.

Editing configuration files

Configuration files are mostly "find the key, change the value". sed does it by regular expression; AWK is better when the value must be calculated.

Bash Comment, uncomment and set values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

To apply to the real file: sed -i.bak 's/^port *=.*/port = 9090/' app.conf writes the change and keeps app.conf.bak.

Extracting pieces from text

grep -o is the simplest extractor, but sed and AWK are better when you need part of a match or a cleaned result.

Bash Emails, URLs, numbers and tags

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Stripping tags with s/<[^>]*>//g is fine for quick text extraction, but it is not an HTML parser: it fails on tags that span lines or contain > inside attributes. Use a real parser for important work.

Converting formats

Bash Separators and shapes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Doing the same edit to many files

Combine a shell loop or find with sed -i. Review first with a plain grep -rl so you know which files will change.

Bash Rename a word across files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

sed -i accepts several files directly (sed -i.bak 's/a/b/g' *.txt), but the loop shows how to limit the edit to files that match. Remember: without a backup suffix the previous contents are gone.

Multi-step surgery with a script

When one-liners start chaining with -e, consider a file of commands run with sed -f, or an AWK program file with -f. They are easier to read and to version control.

Bash A sed script file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Common mistakes

  • Using -i on a file you have not backed up.
  • Greedy matches: s/<.*>// removes from the first < to the last >. Use [^>]*.
  • Forgetting that a, i, c one-liners are GNU specific on some systems.
  • Running sed -i in a loop over * that includes directories or binary files.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which command appends a line after every line matching beta?

    a appends after the matching line, i inserts before it and c replaces it.
  2. 2. Why is s/<.*>//g a poor way to strip tags?

    Greedy matching spans from the first < to the last >. Use [^>]* to stop at the first >.
  3. 3. How do you edit a file in place but keep a backup?

    -i.bak writes the changes to the file and saves the original as file.bak.
  4. 4. What is a safe first step before running sed -i on many files?

    Previewing the files and the substitution output shows exactly what would change.

Summary

  • i, a and c insert, append and replace whole lines; AWK can add headers and footers in BEGIN and END.
  • Config edits: s/^key.*/key = value/, comment with s/^/# /, uncomment with s/^#//.
  • Extract with sed -n 's/.*\(...\).*/\1/p', grep -o, or an AWK match loop.
  • Convert separators with sed 's/\t/,/g' or awk -v OFS=, '{ $1 = $1; print }'.
  • Edit many files with a loop and sed -i.bak; use sed -f or awk -f for longer scripts.

Next: Analyzing Web Server Access Logs