Skip to content
Tutorials Center
My learning

Topics

Lesson 59 of 95 ยท Advanced Bash

Advanced Patterns: Regex, Process Substitution and Coprocesses

Go beyond the basics of Bash: regex matching with BASH_REMATCH, extglob patterns, file descriptors, FIFOs, coproc, lockfiles with flock and parallel jobs with xargs -P and wait -n.

This lesson collects the Bash features that separate "a script that works on my machine" from tools you can lean on: pattern matching that returns captured groups without starting grep or sed, extended globs, extra file descriptors for juggling several inputs and outputs, named pipes and coprocesses for talking to a long-running helper, lockfiles that stop two copies of a script from colliding, and bounded parallelism.

Everything here runs inside the editor. Where a feature is timing-dependent, the examples are arranged so the output is the same on every run (they sort their results, or wait for each job explicitly). The best strategy in your own scripts is the same: parallel code must not depend on who finishes first.

Regex matching with [[ =~ ]] and BASH_REMATCH

The [[ string =~ regex ]] test matches a POSIX extended regular expression (the same dialect as grep -E, see Regular Expressions for the Shell). On success it fills the array BASH_REMATCH: element 0 is the whole match and elements 1, 2, ... are the parenthesised groups. That gives you parsing without any external process.

Bash Parsing with captured groups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The rules that trip people up:

  • Do not quote the regex. [[ $x =~ "a.c" ]] treats the quoted part as a literal string. Put the regex in a variable (re='a.c') or write it unquoted, escaping spaces with a backslash as in the first example.
  • It is ERE only: no \d, \w or \b, and no lazy quantifiers or lookahead. Use [0-9], [[:alnum:]_] and so on. (\s and \w happen to work with glibc, but are not portable.)
  • A match is not anchored: use ^ and $ when you mean the whole string.
  • It matches once. To find every match, loop and cut off the matched text:
Bash Finding all matches in a string

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

For a list of lines, a while read loop with =~ is fine for a few thousand lines; for millions, use awk or grep -o, which are much faster (see Performance, Tricks and Pitfalls).

Extended globs (extglob)

Normal globs cannot express "everything except...". With shopt -s extglob, Bash adds five pattern forms that take a |-separated list inside parentheses:

Pattern Matches
?(a|b) zero or one of the alternatives
*(a|b) zero or more
+(a|b) one or more
@(a|b) exactly one
!(a|b) anything except these

They work in pathname expansion, in case, in [[ == ]], and in ${var#pattern}-style trimming.

Bash extglob in files, case, [[ ]] and trimming

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The classic pitfall is in the comment: extglob changes how Bash parses, so shopt -s extglob on the same line (or inside the same function or if block) as the pattern is too late. Turn it on at the top of the script, and if you define the function in a file that is sourced, enable it before sourcing. rm !(keep.txt) is a favourite one-liner; test it with echo first.

File descriptors beyond 0, 1 and 2

Every open file has a number. 0, 1 and 2 are stdin, stdout and stderr (Standard Streams and Redirection). With exec you can open more, which is how scripts read two files in step, write to a log and a screen at once, or keep a file open across many commands.

TEXT
  exec 3< a.txt        open descriptor 3 for reading a.txt
  exec 4> out.txt      open descriptor 4 for writing out.txt
  read -u 3 line       read one line from descriptor 3
  echo hi >&4          write to descriptor 4
  exec 3<&- 4>&-       close both
  exec {fd}< file      let Bash pick a free number (10 or more) and store it in $fd
Bash Reading two files in lockstep, and writing through a descriptor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Reading from a descriptor with read -u is also the cure for the "ssh eats my loop's stdin" problem from SSH, scp and Working Remotely: loop on descriptor 3 and nothing inside the loop can steal it. Opening a descriptor read-write with exec 3<>file is possible too, and is the trick for named pipes below.

Process substitution, revisited

<(command) gives you a file name whose content is the command's output, so any program that expects files can read the output of pipelines. >(command) does the opposite: a file name that feeds a command's input. Under the hood it is a /dev/fd/N path or a named pipe.

Bash Comparing pipelines, and keeping variables out of subshells

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

A caveat: Bash does not wait for >(...) processes to finish (hence the polling loop above); for anything where completion matters, write to a file and wait for the job explicitly, or use a coprocess.

Named pipes (FIFOs)

A FIFO is a pipe with a name in the filesystem, made with mkfifo. Two unrelated processes can connect through it. The rule that surprises everyone: opening a FIFO blocks until the other end is opened too. That is a feature (the reader waits for the writer) and a deadlock risk in a single script.

Bash A FIFO between a writer job and a reader

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

To avoid blocking on open, open the FIFO read-write (exec 3<>channel), which succeeds at once because the shell itself holds both ends. This is how a long-running script keeps a command queue: other processes append commands to the FIFO, and the script reads them with read -u3.

Coprocesses: a helper you can talk to

coproc starts a command in the background with two pipes attached to it, so your script can write requests and read answers. It is the right tool when you want to keep one helper process alive (a calculator, a database client, a filter) instead of starting it once per request.

The coprocess gets an array NAME with the read end in ${NAME[0]} and the write end in ${NAME[1]}, and the pid in NAME_PID. The helper must write one line per answer and flush it; many programs buffer when writing to a pipe, so use a shell loop or flags such as stdbuf / --line-buffered.

Bash A shouting coprocess

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Two facts to remember. First, Bash 5.0 supports only one coprocess at a time (it warns if you start a second while the first is alive), and its file descriptors are not inherited by ( ... ) subshells. Second, always close the write end so the helper can exit; otherwise your script's wait hangs forever. For several workers, use FIFOs or xargs -P below.

Locking: one instance at a time

Two copies of a backup script that run at once (cron started the next one before the first finished) corrupt data. A lock makes the second copy stop or wait. There are two reliable techniques.

mkdir-based: creating a directory is atomic. Exactly one process can succeed, so it works anywhere and needs no extra tool. Remember to remove the lock on every way out with a trap (see Signals and trap).

flock: the kernel holds an advisory lock on a file descriptor and releases it automatically when the process dies, so a crash never leaves a stale lock behind. This is the preferred way on Linux.

Bash mkdir locks and flock locks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notes: the lock file is just an anchor; it can stay on disk. flock -n means non-blocking, -w 2 waits up to 2 seconds. Locks are advisory: they only work if every contender asks for them. They also do not work reliably across NFS. A common one-line form for cron jobs is flock -n /var/lock/myjob.lock myjob.sh, which runs the command only if the lock is free.

Parallelism: xargs -P, background jobs and wait -n

For independent tasks (compress a hundred files, call an API for a hundred IDs), run several at once, but not all at once. Two patterns cover nearly everything.

xargs -P N runs up to N copies of a command in parallel (xargs, Command Substitution and Process Substitution). The output arrives in completion order, so pipe it through sort when order matters.

A job pool in Bash: start jobs in a loop; when the pool is full, wait -n blocks until any one job finishes (Bash 4.3+) and frees a slot.

Bash Bounded parallelism: xargs -P and a wait -n pool

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

wait -n returns the exit status of the job that ended, but in Bash 5.0 it does not tell you which one (Bash 5.1 adds wait -n -p var). That is why the last example records each $! and waits for them individually. With set -e active, wait -n returning a non-zero status will end your script, so handle it explicitly in pools.

Parallel jobs that write to the same file or the same terminal interleave their output unpredictably. Give each job its own output file, or make each job emit one complete line with a single echo/printf, and combine the results afterwards.

When to use what

Need Use
Parse one line into fields, without external tools [[ =~ ]] and BASH_REMATCH
"All files except..." extglob !(...)
Read two inputs together or keep a file open exec 3<, read -u3
Compare or loop over a pipeline's output <( ... )
Persistent helper process coproc
Prevent overlapping runs flock (or mkdir if flock is missing)
Many independent tasks xargs -P or a wait -n pool
Heavy text work on huge files awk, grep, sort: not Bash loops

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. After [[ "2026-03-14" =~ ^([0-9]+)-([0-9]+)-([0-9]+)$ ]], what is ${BASH_REMATCH[2]}?

    BASH_REMATCH[0] is the whole match and the numbers 1, 2, 3 are the groups in order, so [2] is the month, 03.
  2. 2. Why does [[ $x =~ "a.c" ]] often not behave like a regex?

    Quoted parts of the pattern are matched literally. Keep the regex unquoted or in a variable used without quotes.
  3. 3. You want ls to list everything except .log files. What do you need?

    !(pattern) is an extended glob. Because extglob changes how Bash parses commands, it has to be enabled before the line that uses it.
  4. 4. What is the main advantage of flock over a mkdir lock?

    A flock is tied to an open file descriptor. When the process exits for any reason, the lock goes with it. A mkdir lock must be removed by a trap, and a SIGKILL leaves it behind.
  5. 5. In a job pool, what does wait -n do?

    wait -n returns as soon as the first running job ends, which lets you refill the pool one slot at a time. Plain wait waits for all of them.

Summary

  • [[ s =~ ERE ]] fills BASH_REMATCH (0 = whole match, 1.. = groups). Keep the regex unquoted or in a variable, anchor it yourself, and loop to find several matches.
  • shopt -s extglob enables ?() *() +() @() !() in globs, case, [[ ]] and trimming; enable it before the line that uses it.
  • exec N< file, exec N> file, read -u N, exec {fd}< and exec 6>&1 give you extra descriptors and a way to save and restore stdout.
  • <(cmd) and >(cmd) turn commands into file names; done < <(cmd) keeps the loop in the current shell.
  • A FIFO (mkfifo) blocks on open until both ends exist; coproc gives you a two-way pipe to a helper (close the write end when finished).
  • Use flock (or atomic mkdir) so that only one copy of a script runs; locks are advisory.
  • Bound parallelism with xargs -P N or a wait -n pool, give each job its own output, and collect exit codes by waiting on each PID.

Next: printf, Dates and Formatting