Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 38 ยท Getting Started

Regular Expressions for sed and AWK

Learn the regular expressions that sed and AWK use: literals, anchors, character classes, repetition, groups, alternation, and the difference between BRE and ERE.

Almost every sed command and many AWK patterns contain a regular expression (regex): a compact way to describe a set of strings. [0-9]+ means "one or more digits", ^# means "a line starting with a hash". If you already know regexes from another language, skim this lesson for the differences. If not, take your time: regexes are the engine of both tools.

You will learn the building blocks, the two flavors (BRE and ERE), and the portable subset that works in every sed and AWK. There is also a deeper treatment in the Linux regex lesson.

Literals and the dot

Ordinary characters match themselves. cat matches the three letters c, a, t anywhere in the line. The dot . matches any single character, and the backslash removes a special meaning: \. matches an actual dot.

Bash Literal text, dot and escaped dot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice that c.t matches cat, cot and c.t, but not cart (two characters between c and t).

Anchors

Anchors match a position, not a character:

Anchor Matches
^ the start of the line
$ the end of the line

So ^# finds comment lines, ;$ finds lines ending in a semicolon, and ^$ finds empty lines. In sed's s command, an anchor is how you edit only the beginning or end of lines: s/^/> / adds a prefix and s/$/;/ adds a suffix.

Bash Anchors in action

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Character classes

Square brackets match one character from a set:

  • [abc] matches a, b or c.
  • [a-z] matches a range; [0-9] matches a digit.
  • [^0-9] matches any character that is not a digit (the caret negates when it is first).
  • To include ] put it first ([]x]), to include - put it last ([a-z-]).

POSIX also defines named classes inside brackets: [[:digit:]], [[:alpha:]], [[:space:]], [[:upper:]], [[:lower:]], [[:alnum:]], [[:punct:]]. They work in GNU sed, grep, gawk and the one-true-awk. The mawk used on this site is old and does not support them, so in AWK examples here we write [0-9] and [a-zA-Z] instead; they work everywhere.

Bash Character classes in sed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The second command replaces everything that is neither a digit nor a space. The last one uses GNU's \L to lower-case the match; that is a GNU sed extension, not POSIX.

Repetition

A quantifier says how many times the item before it may repeat:

Quantifier Meaning
* zero or more
+ one or more
? zero or one
, {n,m} exactly n, or between n and m

* works everywhere. The others depend on the flavor, which brings us to the main source of confusion.

BRE versus ERE

POSIX defines two syntaxes:

  • BRE (basic regular expressions): the default in sed and grep. The characters + ? | ( ) { } are plain text; to get their special meaning you write a backslash before them: \+, \?, \|, \(, \), \{, \} (GNU extensions for + ? |).
  • ERE (extended regular expressions): + ? | ( ) { } are special as they are. Used by awk, grep -E, and sed -E (or sed -r in old GNU sed).
TEXT
BRE:  sed 's/\(ab\)\+/X/'        ERE:  sed -E 's/(ab)+/X/'
BRE:  sed 's/a\|b/X/'            ERE:  sed -E 's/a|b/X/'

The rule of thumb for this course: use sed -E whenever you need groups, alternation or +. It reads better and also works on macOS/BSD sed. AWK always uses ERE.

Bash BRE and ERE side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Greedy matching

Quantifiers are greedy: they match as much as they can. In s/<.*>/X/ applied to <b>bold</b>, the .* swallows everything up to the last >. sed has no lazy quantifier, so the standard trick is a negated class: <[^>]*> matches one tag at a time.

Bash Greedy matches and the negated class trick

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first command prints X: one huge greedy match. The second replaces each tag separately and keeps the text between them.

Regex in AWK

In AWK a regex between slashes is a pattern: /error/ { n++ }. The match operators are ~ (matches) and !~ (does not match), used on any string: $3 ~ /^[0-9]+$/. A dynamic regex can also be a string, in which case backslashes must be doubled: $0 ~ "\\.". You will use these heavily in AWK patterns.

Interval expressions {n,m} and bracket classes like [[:digit:]] are POSIX but not supported by every awk. gawk and the one-true-awk accept them; the mawk 1.3.3 behind our runner does not. If your script must run everywhere, repeat the item ([0-9][0-9][0-9]) instead of [0-9]{3}.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which regex matches an empty line?

    ^ marks the start and $ the end; with nothing between them only an empty line matches.
  2. 2. In basic regular expressions (plain sed), how do you write 'one or more of ab' as a group?

    In BRE the grouping characters and + need a backslash. Use sed -E to write (ab)+ without them.
  3. 3. What does <[^>]*> match in <b>bold</b> when used with the g flag?

    [^>]* cannot cross a >, so each match stops at the first closing bracket, giving two matches.
  4. 4. Which operator tests a regex against a string in AWK?

    AWK uses ~ (and !~ for no match), for example $1 ~ /^a/.

Summary

  • Literals match themselves; . matches any character; \ escapes.
  • ^ and $ anchor to the start and end of the line.
  • [...] matches one character from a set; [^...] negates it.
  • *, +, ?, {n,m} repeat; BRE needs backslashes for most of them, ERE (sed -E, awk) does not.
  • Matching is greedy; use negated classes to stop early.

Next: Versions and Portability