Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 43 ยท Regular Expressions

Advanced Regex

Go beyond the basics in Perl regex: lookahead and lookbehind, backreferences, possessive quantifiers, recursion, qr// objects and how to avoid slow patterns.

Perl's regex engine is famous for a reason: it supports features that many other tools do not. Lookaround checks the text next to a position without using it up. Backreferences match the same text twice. Possessive quantifiers stop wasteful backtracking, and recursion lets one pattern match nested structures such as balanced parentheses.

This lesson covers those tools, shows how to store patterns with qr//, and ends with the performance traps that make a pattern take forever. You should already know groups and modifiers from Captures and Modifiers.

Perl is not installed on this site, so the Perl code is read-only. The interactive examples are Python stand-ins. Python's re supports lookaround, backreferences, possessive quantifiers and atomic groups (since Python 3.11) with almost the same syntax, but it has no recursion, so that part is shown with a small function instead.

Lookahead and lookbehind

A lookaround is a zero-width assertion: it checks what comes before or after the current position, but the checked text is not part of the match.

Syntax Name Succeeds when
(?=...) positive lookahead the text ahead matches
(?!...) negative lookahead the text ahead does not match
(?<=...) positive lookbehind the text behind matches
(?<!...) negative lookbehind the text behind does not match
PERL
use v5.36;

my $text = 'Total: $120, tax: $9.50';
my @amounts = $text =~ /(?<=\$)(\d+(?:\.\d\d)?)/g;
say "@amounts";                       # 120 9.50

say "foo" =~ /foo(?!bar)/ ? "ok" : "no";       # ok
say "foobar" =~ /foo(?!bar)/ ? "ok" : "no";    # no

The dollar sign is not in the results, because the lookbehind only looks at it. Lookaheads are also the standard trick for checking several conditions at once. Each lookahead starts at the same spot and tests one rule:

PERL
my $strong = qr/^(?=.*\d)(?=.*[a-z])(?=.*[A-Z]).{8,}$/;
say "Passw0rdOK" =~ $strong ? "strong" : "weak";    # strong
say "password"   =~ $strong ? "strong" : "weak";    # weak

Two practical notes. Keep lookbehinds simple: they traditionally need a fixed length, and although some recent Perl releases allow limited variable length, code that relies on it is less portable. And when you want "match this, but only report what follows", Perl offers \K ("keep"), which drops everything matched so far from the result and works nicely with s///:

PERL
(my $s = "price=100") =~ s/price=\K\d+/200/;
say $s;                               # price=200

Backreferences

A backreference matches the same text a group matched earlier. \1 is the first group, \2 the second. Because \1 followed by digits can be ambiguous, \g{1} is clearer, and \g{-1} means "the group just before this point". For named groups use \k<name>.

PERL
my $dup = "this is is a test test";
my @repeats = $dup =~ /\b(\w+)\s+\1\b/g;
say "@repeats";                       # is test

(my $fixed = $dup) =~ s/\b(\w+)(?:\s+\1\b)+/$1/g;
say $fixed;                           # this is a test

say "abba" =~ /^(\w)(\w)\g{2}\g{1}$/ ? "palindrome" : "no";   # palindrome

Remember that in the replacement part of s/// you write $1, not \1. The backslash form belongs inside the pattern.

Python Lookaround and backreferences (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Possessive quantifiers and atomic groups

Normally, when a pattern fails later on, the engine backtracks: it gives back characters that a quantifier had taken and tries again. Often that is exactly what you need, but sometimes it only wastes time.

A possessive quantifier (*+, ++, ?+, {n,m}+) never gives anything back. An atomic group (?>...) does the same for a whole sub-pattern: once it has matched, the engine will not re-enter it to try other splits.

PERL
say "aaaa" =~ /a+a/  ? "match" : "no match";    # match  (backs off one a)
say "aaaa" =~ /a++a/ ? "match" : "no match";    # no match (a++ keeps all four)

Possessive matching is a speed tool as much as a logic tool: when you know a piece of the pattern can never usefully shrink, say so, and failures become fast. A good example is [^"]*+ inside a quoted string, because the closing quote can never be part of [^"].

Advanced features, one job each

Lookaround Check the neighbors without using them up.
Backreference Match the same text again.
Possessive Never give characters back.
Recursion A pattern that calls itself.

Reach for these only when a simple pattern cannot do the job.

1 / 2

Recursion

Perl patterns can call themselves. (?R) re-runs the whole pattern, (?1) re-runs group 1, and (?&name) calls a named group. This is how a regex can match nested, balanced structures, which ordinary patterns cannot do:

PERL
my $balanced = qr/^(\((?:[^()]++|(?1))*\))$/;

for my $t ("(a(b)c)", "((x)", "()") {
    say "$t: ", ($t =~ $balanced ? "balanced" : "not balanced");
}
TEXT
(a(b)c): balanced
((x): not balanced
(): balanced

Read the pattern aloud: an opening parenthesis, then any number of items that are either a run of non-parenthesis characters or another balanced group ((?1) is a call to group 1 itself), then a closing parenthesis.

For bigger grammars, define named rules inside (?(DEFINE)...) and call them with (?&name). Recursive patterns are powerful, but also hard to read. For real parsing, such as JSON or program source, use a proper parser module from CPAN instead.

Code blocks: a warning

Perl also allows (?{ code }) inside a pattern, which runs Perl code during the match. It is useful for debugging and for rare tricks, but it makes patterns hard to read, can slow matching, and is dangerous with outside data. Perl refuses to run code from a pattern that is built from a string variable unless the program says use re 'eval', and you should never enable that for patterns that came from users, files or a network. A pattern is code when it can run code, so treat it with the same caution.

Also never put raw user input into a pattern. Wrap it in quotemeta or \Q...\E so special characters lose their meaning:

PERL
my $needle = "a.b*c";
say "xa.b*cx" =~ /\Q$needle\E/ ? "found" : "missing";   # found
say "xaXbbbc" =~ /\Q$needle\E/ ? "found" : "missing";   # missing

Precompiled patterns with qr//

qr// compiles a pattern once and returns an object you can store, pass around and embed in other patterns. Modifiers given to qr stay attached to it:

PERL
my $word  = qr/[a-z]+/i;
my $pair  = qr/($word)=($word)/;

if ("Color=Red" =~ $pair) {
    say "$1 is $2";                   # Color is Red
}
say $word;                            # (?^i:[a-z]+)

Printing a qr object shows the pattern wrapped as (?^flags:...). Interpolating it into a bigger pattern keeps its flags intact, which makes it easy to build complex patterns from small named parts. This is also the right way to match any of a list of words:

PERL
my @words = ("cat", "c.t", "dog");
my $alt   = join "|", map { quotemeta } @words;
my $re    = qr/\b(?:$alt)\b/;
say join ",", "a cat, a c.t and a dog" =~ /($re)/g;   # cat,c.t,dog

Compile outside loops. A qr object built once and reused is faster than a variable pattern that Perl has to recompile each time it changes. (Old advice about the /o modifier is obsolete: use qr// instead.)

Python Possessive, recursion stand-in and compiled patterns (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Performance pitfalls

Most patterns run in microseconds, but a few shapes can take longer than you will live. The cause is catastrophic backtracking: nested or overlapping quantifiers create exponentially many ways to split the same text, and when the overall match fails, the engine tries them all.

The textbook example is (a+)+b against a long run of a with no b. Each extra a roughly doubles the work. Perl has some defenses and catches certain simple cases, but you cannot count on it for every pattern, so write patterns that cannot blow up:

  • Be specific. [^,]* is better than .* when you mean "up to the next comma".
  • Avoid nested quantifiers over the same characters, such as (\w+\s*)+$. Rewrite them, or make the inner part possessive: (?>\w+).
  • Anchor when you can, with ^, \A or a leading literal. A pattern that starts with a fixed word lets Perl skip straight to places where that word occurs.
  • Do not start with .* unless you must. Prefer anchored or specific starting parts.
  • Use plain string functions for fixed text. index($s, "needle") beats a regex for a literal search.
  • Compile once with qr// and reuse it (see above).
  • Never run patterns from untrusted sources against large inputs. A hostile pattern or input is a denial-of-service risk.

To measure rather than guess, use the Benchmark module (see Performance and Profiling).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does (?<=\$)\d+ match?

    (?<=...) is a positive lookbehind: it checks the text before the position but does not consume it.
  2. 2. Which pattern fragment matches a word repeated twice in a row, such as the the?

    \1 is a backreference to the text captured by group 1. $1 is a variable and does not work inside the pattern for this purpose.
  3. 3. Why does "aaaa" =~ /a++a/ fail?

    A possessive quantifier refuses to backtrack. After a++ has eaten every a, nothing is left for the last a in the pattern, so the match fails.
  4. 4. What is the safe way to put user input into a pattern?

    quotemeta and \Q...\E escape metacharacters so the input is matched as literal text and cannot change the pattern.
  5. 5. What does qr/\d+/ give you?

    qr// compiles a regex and returns an object that can be matched against, stored in a variable and embedded in larger patterns.

Summary

  • Lookaround ((?=), (?!), (?<=), (?<!)) checks neighbors without consuming them; combine several lookaheads to test multiple rules.
  • Backreferences (\1, \g{-1}, \k<name>) match the same text again; use $1 only in the replacement and in code.
  • Possessive quantifiers and atomic groups never backtrack, which prevents certain failures from being slow.
  • (?R), (?1) and (?&name) allow recursion for nested structures; for large grammars, use a real parser.
  • Never interpolate untrusted data into a pattern without quotemeta/\Q\E, and avoid (?{ }) with outside data.
  • qr// compiles once and composes well; avoid nested quantifiers and leading .* to keep patterns fast.

Now that you can transform text in memory, learn to work with real files in Reading and Writing Files.