Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 38 ยท AWK Programs

String Functions and Regular Expressions in AWK

Master AWK's text functions: length, substr, index, split, sub, gsub, match with RSTART and RLENGTH, sprintf, toupper and tolower, plus regex matching rules.

Fields and arrays get you far, but real data is messy: padded values, dates glued together, prefixes to strip, tags to pull out of a line. AWK has a compact set of string functions and a built-in regular expression engine for exactly that. Learn these ten functions and you can reshape almost any line of text.

All the examples run in the editors below and use only features that every common AWK shares (POSIX extended regexes without {n,m} intervals, because some small AWKs omit them).

Regex matching operators

A regular expression written between slashes is a pattern for a whole line. For anything else, use the match operators:

AWK
/error/               # the current line matches
$2 ~ /^[0-9]+$/       # field 2 is all digits
$1 !~ /^#/            # field 1 does not start with #
$0 ~ pattern_string   # the right side can be a string (a "dynamic regex")

When a regex is held in a string, remember that backslashes need doubling: "\\." is the regex \., a literal dot.

length, substr, index

  • length(s) is the number of characters (length alone means length($0)).
  • substr(s, start, n) returns n characters from position start. Positions start at 1. Omit n for "to the end".
  • index(s, t) is the position of the first t in s, or 0 if absent.
Bash Slicing strings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

split: one string into an array

split(s, arr, sep) cuts s at each sep (a regex, or a single-space string meaning "runs of whitespace"), fills arr[1]..arr[n] and returns n. It clears arr first.

Bash Splitting dates and paths

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice that a leading / gives an empty p[1], and that the separator can be a regex.

sub and gsub: search and replace

sub(regex, replacement, target) replaces the first match; gsub replaces all. If you omit target it works on $0. Both return the number of replacements and change the target in place. Inside the replacement, & stands for the matched text; write \\& for a literal ampersand.

Bash sub and gsub

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last example shows a subtle side effect: assigning to a field (which sub does with target $2) makes AWK rebuild $0 with OFS between fields.

match, RSTART and RLENGTH

match(s, regex) returns the start position of the leftmost longest match (or 0) and sets RSTART and RLENGTH. Combined with substr it extracts text, which plain sub cannot do easily.

Bash Extracting with match

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The while (match(...)) loop is the standard way to walk all matches in a line, since portable AWK has no "find all" function.

Case, sprintf and building strings

toupper(s) and tolower(s) change case. sprintf(fmt, ...) formats like printf but returns the string, which is how you pad, round or zero-fill before storing a value. Concatenation is just writing values next to each other.

Bash Case and formatting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Putting it together

flowchart LR
  A["raw line"] --> B["split / substr<br/>cut it up"]
  B --> C["sub / gsub<br/>clean it"]
  C --> D["match<br/>extract"]
  D --> E["sprintf<br/>format"]

A typical cleaning pipeline inside one AWK action.

Common mistakes

  • Starting substr at 0: positions start at 1.
  • Forgetting that sub and gsub modify the target, so they do not "return the new string".
  • Writing \. inside a string for a dynamic regex: it needs "\\.".
  • Using {n,m} intervals on an AWK that does not support them: repeat the item instead, as in [0-9][0-9][0-9].

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does substr("abcdef", 2, 3) return?

    Positions start at 1, so starting at 2 for 3 characters gives "bcd".
  2. 2. What is the difference between sub and gsub?

    sub stops after the first replacement; gsub replaces all non-overlapping matches and returns how many it made.
  3. 3. After a successful match(s, re), how do you get the matched text?

    match sets RSTART (start position) and RLENGTH (length); substr then cuts out the text.
  4. 4. In a replacement string, what does & mean?

    & stands for the text matched by the regex. Write \& to get a literal ampersand.

Summary

  • length, substr (1-based), index and split take strings apart.
  • sub replaces the first match, gsub all of them, both in place; & means the match.
  • match plus RSTART/RLENGTH plus substr extracts text.
  • toupper, tolower and sprintf change case and format values.
  • Dynamic regexes in strings need doubled backslashes.

Next: printf and Formatted Reports in AWK