Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 38 ยท AWK Basics

AWK String and Math Functions

Use AWK's built-in functions: length, substr, index, split, sub, gsub, match with RSTART and RLENGTH, tolower, toupper, sprintf, and math functions such as int, sqrt, rand.

AWK comes with a small but complete set of built-in functions for strings and numbers. They are all part of POSIX, so they work in every awk. With them you can cut, search, replace and normalise text inside fields, which is what turns AWK from a "column picker" into a text-processing language. This lesson goes through each function with runnable examples.

length

length(s) returns the number of characters of a string. Without arguments it means length($0), and you may even drop the parentheses: length alone is the same.

Bash length

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

substr

substr(s, start [, len]) returns part of a string. Positions start at 1. If len is omitted you get the rest of the string. A start below 1 is treated as if the string had spaces in front, so be careful with 0.

Bash substr

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The date example turns 20240517 into 2024-05-17 by slicing. Slicing by fixed columns is also the way to read fixed-width files.

index

index(s, t) returns the position of the first occurrence of the plain string t in s, or 0 if absent. It is not a regex search, so special characters need no escaping: it is the right tool for literal text such as index($0, "a.b").

Bash index

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The second command splits on the first equal sign only, which -F= cannot do. That is a common use of index plus substr.

tolower and toupper

Bash Case conversion and case-insensitive matching

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

POSIX AWK has no case-insensitive flag, so tolower($0) ~ /pattern/ is the portable method. The last command capitalizes the first letter.

split

split(s, arr, sep) cuts s at the separator sep (a regex or a single character; if omitted, FS is used) and stores the pieces in arr[1], arr[2], ...; it returns the number of pieces. It clears arr first. Arrays are covered in detail later in the course, but you can use split already:

Bash split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

A string separator longer than one character, and split(s, a, "") (split into characters), behave differently between awks, so prefer single-character or regex separators.

sub and gsub

sub(regex, replacement [, target]) replaces the first match; gsub(...) replaces all matches. The target defaults to $0; assigning to $0 or a field recomputes the other. Both return the number of replacements. In the replacement, & stands for the matched text (write \\& for a literal ampersand).

Bash sub and gsub

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The trim idiom gsub(/^ +| +$/, "") removes leading and trailing blanks in one pass. Modifying a field with sub(..., $1) rebuilds $0 with OFS. POSIX awk has no back-references in the replacement (\1); that is the main job of gawk's gensub. To rearrange pieces, use match with substr, or split.

match, RSTART and RLENGTH

match(s, regex) returns the position where the first match begins (or 0) and sets RSTART (same value) and RLENGTH (length of the match, -1 if none). Together with substr you can extract the matched text:

Bash Extracting with match

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The second command finds all e-mail-like strings by looping: after each match, it continues with the rest of the string. This is the standard POSIX way to extract all matches (gawk's third argument to match is an extension).

sprintf

sprintf(format, ...) formats like printf but returns the string (see printf). Use it to pad, round, or build keys:

Bash sprintf for formatting values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Math functions

Function Result
int(x) integer part (truncates toward zero)
sqrt(x) square root
exp(x), log(x) e to the x, natural logarithm
sin(x), cos(x) radians
atan2(y, x) arctangent of y/x
rand() pseudo-random number in [0, 1)
srand([seed]) set the seed; returns the previous seed
Bash Math functions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice that AWK has no log10 and no abs function; you write log(x) / log(10) and x < 0 ? -x : x.

Random numbers

rand() always returns the same sequence unless you call srand() first, because the default seed is fixed. srand() with no argument seeds from the time of day, giving different numbers on each run. Because the exact numbers depend on the implementation, tests and tutorials only rely on their range and count:

Bash rand and srand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

int(rand() * n) gives an integer from 0 to n-1, and adding 1 makes a die.

gawk-only extras

You will see these in tutorials: gensub, strftime, systime, asort, asorti, toupper on UTF-8, length(array), patsplit, and/or/xor, tolower with locale rules. They are not POSIX. Check your awk with awk --version; if you want portable code, write the equivalent with the functions above (see Versions and Portability).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does substr("abcdef", 2, 3) return?

    Positions start at 1: three characters starting at position 2 are b, c, d.
  2. 2. What does gsub(/a/, "b") return?

    sub and gsub modify the target and return how many substitutions they did.
  3. 3. After match($0, /[0-9]+/) succeeds, which expression gives the matched text?

    match sets RSTART and RLENGTH; substr extracts the text between them.
  4. 4. How do you compare ignoring case in portable AWK?

    IGNORECASE and the i flag are gawk/GNU features; lower-casing first works everywhere.
  5. 5. Why does a program calling rand() print the same numbers every run?

    Without srand, the generator starts from the same seed each run.

Summary

  • length, substr (1-based), index (plain text), tolower/toupper, split cover most string work.
  • sub and gsub replace using regexes and return a count; & is the match, and there are no back-references in POSIX awk.
  • match plus RSTART/RLENGTH plus substr extracts text; loop to extract all matches.
  • int, sqrt, exp, log, sin, cos, atan2, rand, srand are the math tools; rand needs srand to vary.
  • Avoid gawk-only functions if the script must run everywhere.

Next: AWK Associative Arrays: Counting, Grouping and Summing