Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 38 ยท Regex in Languages and Tools

Regex on the Command Line

Use regular expressions in the shell with grep, sed and awk. Learn basic versus extended syntax, POSIX classes, common options, and recipes for filtering, extracting and rewriting log and text files.

Regular expressions started in Unix text tools, and they are still the fastest way to slice through a log file or rewrite a hundred config files at once. Three programs cover most needs: grep finds lines, sed rewrites lines and awk treats lines as records with fields. They all use POSIX-style regex, which is close to what you know but with a few important differences.

Every example here is a real shell script that runs below, so you can change a pattern and watch the output. Each script creates its own sample file first.

Basic and extended syntax

POSIX has two dialects. Basic regular expressions (BRE) are the default for grep and sed: the characters ? + | ( ) { } are literal unless you put a backslash before them (\+, \(). Extended regular expressions (ERE) treat them as operators like the syntax you already know. Use grep -E (or egrep) and sed -E for ERE; it is almost always what you want. awk always uses ERE.

Task BRE ERE
One or more digits [0-9]\+ [0-9]+
Group \(ab\)\{2\} (ab){2}
Literal plus + \+

Alternation follows the same rule: GNU grep and sed accept a backslash followed by a pipe as the BRE alternation operator, while ERE uses a plain pipe.

Also remember: \d is not part of POSIX. Use [0-9] or the POSIX classes [[:digit:]], [[:alpha:]], [[:alnum:]], [[:space:]], [[:upper:]], [[:lower:]], [[:punct:]]. Note the double brackets: the outer pair is the bracket expression, the inner is the class name.

grep: filter lines

Useful options: -E extended syntax, -i ignore case, -v invert, -o print only the matching part, -n line numbers, -c count, -w whole words, -x whole line, -r recurse.

Bash grep recipes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

grep -P enables Perl-compatible syntax (\d, lookahead, lazy quantifiers) on GNU grep, but it is not available everywhere (macOS's grep lacks it), so prefer -E in scripts you share.

sed: rewrite lines

sed applies commands to each line. The substitute command is the one you will use: s/pattern/replacement/flags. In the replacement, & stands for the whole match and \1...\9 for groups. Flags: g all matches on the line, i ignore case (GNU), p print. Use -n with p to print only changed lines, and -E for extended syntax.

Bash sed recipes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

When the pattern contains slashes, pick a different delimiter (s|a|b|, s#a#b#) instead of escaping every slash. Be careful with sed -i (edit in place): GNU sed takes -i alone, while macOS needs -i '', so test without -i first and make a backup.

awk: fields and regex together

awk runs a pattern-action program for each line. A regex between slashes is a pattern, and ~ and !~ test a field against a regex. Fields are $1, $2, ... and $0 is the whole line.

Bash awk recipes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Quoting and shell traps

  • Put patterns in single quotes so the shell does not expand $, *, ! or backticks.
  • To put a single quote inside a single-quoted pattern, close and reopen: 'it'\''s'.
  • The shell's globbing (*.txt) is a different, simpler language from regex. * in a glob means "anything", in a regex "repeat the previous item".
  • Anchors ^ and $ in sed and grep match at the ends of each line, since these tools process one line at a time. To match across lines you need sed -z (GNU), awk with a custom record separator, or perl -0.
  • Locale matters: in some locales [a-z] also matches uppercase letters, and . can match multi-byte characters. Set LC_ALL=C for byte semantics.

Which tool when

Goal Tool
Which lines match? How many? grep
Replace or delete text sed
Columns, sums, per-field rules awk
Complex multi-line or PCRE features perl -pe, ripgrep (rg), or a script in a language you know

The patterns you already know from the JavaScript API and other languages carry over; mostly you swap \d for [0-9] and remember -E.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use grep -E instead of plain grep for most patterns?

    Plain grep uses basic regex where those characters are literal. -E selects extended syntax, which matches what most people expect.
  2. 2. Which is the POSIX way to match a digit?

    \d is a Perl-style shorthand. POSIX uses bracket expressions such as [0-9] or the class [[:digit:]].
  3. 3. What does sed -E 's/[0-9]+/<&>/g' do to the line a1b22?

    & is the whole match and g replaces every match on the line, so each run of digits is wrapped.
  4. 4. Why should patterns be placed in single quotes in the shell?

    The shell interprets many characters inside double quotes; single quotes pass the pattern through untouched.

Summary

  • grep selects lines, sed rewrites them with s/pattern/replacement/, awk combines regex and fields.
  • POSIX has BRE (backslashed operators) and ERE (-E, plain operators); there is no \d, use [0-9] or [[:digit:]].
  • &, \1 and so on reuse matches in sed replacements; ~ tests a field in awk.
  • Quote patterns with single quotes, choose another delimiter for paths, and mind line-by-line processing.
  • When you need lookaround or multi-line matching, reach for grep -P, perl or a script.

Next: Project: A Log Analyzer