Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 95 ยท Working with Text

Regular Expressions for the Shell

Learn regular expressions for the Linux shell: basic vs extended syntax, anchors, quantifiers, groups, backreferences, POSIX classes and recipes for grep and sed.

A regular expression (regex) is a tiny language for describing patterns of text. Instead of searching for the exact word error, you can describe "a date like 2026-03-01", "an email-looking word" or "a line that starts with a number and ends with a semicolon". Regexes power grep, sed, awk, vim, programming languages and countless other tools. Learn them once and you will use them for the rest of your career.

Two warnings before we begin. First, a regex is not a glob: *.txt in a filename pattern means "anything ending with .txt", but in a regex * means "repeat the previous item" (see Wildcards and Globbing). Second, regex dialects differ. This lesson teaches the two dialects built into the standard Linux tools, basic (BRE) and extended (ERE), plus Bash's own [[ =~ ]] operator, and tells you where they differ from the Perl-style regexes of Python or JavaScript.

Two dialects: BRE and ERE

POSIX defines two flavours. They have the same ideas but different spelling, which is the most confusing part for beginners.

Feature Basic (BRE) Extended (ERE)
Tool option default for grep, sed grep -E, sed -E (also awk)
One or more \+ (GNU) +
Zero or one \? (GNU) ?
Alternation | (GNU) `
Group \( \) ( )
Interval \{2,4\} {2,4}
Literal +, ?, ( plain characters need a backslash

The rule of thumb: use ERE (-E) unless you have a reason not to. It reads closer to what you will write in other languages, and you do not have to backslash operators. The \+, \? and \| forms in BRE are GNU extensions and do not work on every system.

Building blocks

These parts work in both dialects.

  • Literal characters match themselves: cat matches the text "cat".
  • . matches any single character (except a newline in most tools).
  • Anchors: ^ matches the start of the line, $ the end. ^$ is an empty line.
  • Bracket expressions: [abc] matches one of a, b, c; [a-z] a range; [^0-9] anything but a digit. Inside brackets most special characters lose their meaning, so [.] is a literal dot. To include ] put it first ([]x]), and - last ([a-z-]).
  • POSIX classes inside brackets: [[:digit:]], [[:alpha:]], [[:alnum:]], [[:upper:]], [[:lower:]], [[:space:]], [[:punct:]], [[:xdigit:]]. These respect your locale and are the portable way to say "a letter".
  • Escape: a backslash makes a special character literal: \. is a dot, \$ a dollar sign, \\ a backslash.
Bash Building blocks with grep

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Quantifiers: how many times

A quantifier follows an item (a character, a bracket expression or a group) and says how often it may repeat.

Quantifier (ERE) Meaning
* zero or more
+ one or more
? zero or one
exactly 3
{2,4} between 2 and 4
{2,} 2 or more

So [0-9]+ is "a number", colou?r matches both color and colour, and [0-9]{4} is "four digits". In BRE write [0-9]\+ and [0-9]\{4\}.

Bash Quantifiers and alternation with grep -E

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last two commands show the dialect switch clearly: in BRE the pattern a+b looks for a literal plus sign, while a\+b means "one or more a's followed by b".

Groups, alternation and backreferences

Parentheses group items so a quantifier or an alternation applies to all of them: (ab)+ matches ab, abab, ababab. Groups also capture what they matched, and you can refer to the capture later with \1, \2 and so on. That is a backreference.

  • In a search: ([a-z]+) \1 finds a repeated word, like "the the".
  • In sed replacement text: s/\(a\)\(b\)/\2\1/ swaps two parts, and & stands for the whole match.
Bash Groups and backreferences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

\b (GNU) is a word boundary; it keeps is is from matching the end of this is. Backreferences are a feature of POSIX regexes that goes beyond what a strict "regular language" can do, and they can be slow on huge inputs, so use them where they help.

Greedy matching

Quantifiers are greedy: they match as much as they can while still letting the whole pattern succeed. This regularly surprises people when they try to pull out something between delimiters.

Bash Greedy versus careful patterns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

POSIX regexes have no lazy quantifier (like .*? in Perl or Python). The standard answer is a negated class: "any run of characters that are not the closing delimiter". Learn this idiom; it is the answer to most "match up to X" questions.

The leftmost match wins, and among the matches starting at that spot POSIX prefers the longest. That is why <.*> consumed everything from the first < to the last >.

Special sequences in GNU tools

GNU grep and sed add shortcuts that are handy but not portable to BSD/macOS:

Sequence Meaning
\b word boundary
\< / \> start / end of a word
\w / \W word character ([A-Za-z0-9_]) / its opposite
\s / \S whitespace / non-whitespace
\d not supported in POSIX regexes: use [0-9] or [[:digit:]]

If you need \d, lookahead or lazy quantifiers, use grep -P (GNU only, Perl-compatible) or switch to Python or Perl. Many tutorials show \d with grep -E; it does not work there.

Common recipes

Regexes for real-world formats are always a trade-off between strict and simple. The examples below are good enough for log files and quick filtering, not for security validation.

Bash Recipes: IPs, dates, emails, hex colors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The first IPv4 pattern accepts 10.0.0.256, which is not a valid address; the last one rejects it by spelling out the 0-255 rule. Notice how a regex stored in a variable ($octet) lets you assemble a long pattern from named parts, which is the most readable way to write big expressions in a script.

Regex in Bash: [[ =~ ]]

Bash can match regexes without any external program, which is perfect for validating input in a script. The right-hand side of =~ is an ERE. A successful match fills the array BASH_REMATCH: element 0 is the whole match and 1, 2, ... are the captured groups. (A fuller treatment is in Advanced Patterns.)

Bash [[ =~ ]] and BASH_REMATCH

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last pair is the classic Bash gotcha: quoting the right side turns the regex into plain text. Store the pattern in a variable and use it unquoted. [[ ]] is only in Bash (and zsh/ksh), not in POSIX sh.

Escaping in the shell: the two layers

Writing regexes in a script means two parsers handle your backslashes: first the shell, then the regex engine. Single quotes protect the pattern from the shell completely, so you write the regex as it should look to the tool:

Terminal
grep -E '^[0-9]+\.[0-9]+$' file      # good: the shell passes the pattern untouched
grep -E "^[0-9]+\.[0-9]+$" file      # risky: $ and backslashes get shell treatment

Use double quotes only when you must insert a variable, and then think about which characters the shell will interpret ($, backticks and \).

Choosing a tool

  • Quick search: grep (BRE) or grep -E.
  • Substitution: sed -E 's/pat/repl/' (see sed).
  • Per-field matching: awk '$3 ~ /regex/' (see awk).
  • Validation in a script: [[ $x =~ $re ]].
  • Anything needing \d, lookahead, lazy matching or named groups: grep -P, Perl or Python.
  • Structured formats (JSON, HTML, CSV with quotes): a real parser. Regexes are the wrong tool for nested or quoted structures.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which grep command matches both color and colour?

    In ERE (-E) the ? makes the u optional. In plain BRE, a bare ? is a literal question mark, and -F matches the text literally. The last pattern would also match colouur.
  2. 2. What does sed -E 's/<.*>/X/' produce for <b>bold</b>?

    The quantifier is greedy: .* runs as far as it can, up to the last >, so the whole string is replaced. Use <[^>]*> with the g flag to replace each tag.
  3. 3. Which pattern is the portable way to match one digit in grep, sed and awk?

    POSIX regexes have no \d. Use a range or the POSIX class; \d works only in Perl-style engines such as grep -P.
  4. 4. In [[ $input =~ "$re" ]] the regex never matches as expected. Why?

    Quoting the pattern in [[ =~ ]] disables regex meaning. Put the regex in a variable and write [[ $input =~ $re ]].
  5. 5. What does the group in ([a-z]+) \1 let you find?

    \1 refers back to whatever the first group matched, so the pattern matches a word followed by a space and the identical word, like 'the the'.

Summary

  • Regexes describe text patterns; they are not globs. * repeats the previous item.
  • Grep and sed use BRE by default (\+, \?, \|, \( \)); -E selects ERE (+, ?, |, ( )). Prefer ERE.
  • Building blocks: literals, ., ^, $, bracket sets, POSIX classes like [[:digit:]], and quantifiers * + ? {n,m}.
  • Groups capture text for backreferences (\1); sed can rearrange with them, and & is the whole match.
  • Quantifiers are greedy: use a negated class like [^>]* instead of a lazy match.
  • Bash's [[ $s =~ $re ]] uses ERE and fills BASH_REMATCH; keep the regex unquoted in a variable.
  • Quote regexes in single quotes; use a real parser for JSON, HTML or quoted CSV.

Next: sed: The Stream Editor