Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 95 ยท Working with Text

awk: Column Processing and Reports

Learn awk on Linux: fields, -F, patterns and actions, BEGIN and END, variables, arrays, printf, built-in functions, sums by key and formatted reports.

awk is a small programming language built for one job: reading text record by record (usually line by line), splitting each record into fields (usually columns), and acting on them. If grep finds lines and sed rewrites them, awk understands their structure. It can print the third column, add up a column of numbers, count how often each value appears, and produce a neat report, all in one short command.

The name comes from its authors Aho, Weinberger and Kernighan. There are several implementations: gawk (GNU), mawk, the BSD/macOS awk, and the "one true awk". In our sandbox awk is mawk 1.3.3, a fast and minimal version, so every example here sticks to features that all of them share. Where gawk offers more, the text says so.

The shape of an awk program

An awk program is a list of pattern { action } pairs. For each input line awk tests every pattern, and runs the action of those that match.

TEXT
awk 'pattern { action }  pattern { action } ...' file
  • If you leave out the pattern, the action runs for every line.
  • If you leave out the action, the default is { print }, which prints the line. So awk '/error/' works like grep error.
  • The program is usually in single quotes so the shell does not touch $1, { and so on.

Fields: $1, $2, $NF

awk splits each line into fields at runs of whitespace (spaces and tabs) by default. $1 is the first field, $2 the second, $0 the whole line. The built-in variable NF holds the number of fields, so $NF is the last one. NR is the number of the current record (line number).

Bash Fields, NF and NR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

awk handles irregular spacing for free, because whitespace runs are one separator: apple 3 1.20 has exactly three fields. print a, b puts a space (the output field separator, OFS) between items, while print a b (no comma) concatenates them without any separator. Mixing those up is a very common slip.

You can also change fields: assigning $2 = "x" rebuilds the line with OFS between the fields, and $1 = $1 is the classic trick to normalise spacing.

Field separators: -F and FS

For CSV, /etc/passwd and other formats, tell awk what separates fields with -F. The separator can be a single character or a regular expression.

Bash -F and FS

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

-F, is perfect for simple CSV. As with cut, it cannot understand commas inside quoted fields ("Smith, John"); real CSV needs a CSV-aware tool, or gawk's FPAT. Also remember that -F'|' means "a regex that is a vertical bar", which for a single character is treated literally; for patterns such as . write -F'[.]'.

Patterns: choosing which lines

A pattern before the action decides when it runs. Patterns can be:

Pattern Runs the action when
/regex/ the line matches the regex
$2 ~ /regex/ field 2 matches (!~ for not matching)
$3 > 100 a comparison is true (numbers or strings)
NR == 1, NR > 1, NR % 2 == 0 by line number
expr1 && expr2, expr1 || expr2, !expr combinations
/start/, /end/ a range between two matches
BEGIN, END before the first line / after the last
Bash Patterns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The final two lines show how awk decides what a comparison is: fields that look like numbers compare as numbers, but concatenating "" forces text comparison, where "10" sorts before "9". Unquoted /regex/ is a pattern; a string on the right of ~ can also be a regex stored in a variable.

BEGIN, END and variables

BEGIN { ... } runs once before any input is read; use it to set FS or OFS, initialise counters, or print a header. END { ... } runs once after the last line, where totals are printed. Variables need no declaration: they spring into existence, start as 0 (or the empty string), and are numeric or text depending on use.

Bash Totals with BEGIN and END

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Use -v name=value to hand shell variables to awk. Never splice a shell variable into the program text (awk "$2 > $limit"), because quoting breaks and injection becomes possible. Notice also that sum / NR divides: awk numbers are floating point, so 7 / 2 is 3.5, unlike shell arithmetic.

Control flow and printf

awk's syntax is C-like: if (cond) ... else ..., while, for (i = 1; i <= NF; i++), next (skip to the next line), exit, and ; or newlines between statements. printf works like the C and shell version, which makes aligned reports easy; unlike print, it does not add a newline.

Bash Loops, if and printf

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Unlike the shell, awk does floating-point arithmetic, so $1 / 3 has decimals. Use int() to truncate. Strings and numbers convert automatically, so "3" + 4 is 7; a non-numeric string counts as 0.

Arrays: counting and grouping

awk arrays are associative: the index can be any string. This single feature is what makes awk great at reports: "count by key" and "sum by key" are just count[$1]++ and sum[$1] += $2. Walk an array with for (key in array). The order of that loop is unspecified, so pipe the result through sort when you need stable output.

Bash Count and sum by key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

!seen[$0]++ removes duplicate lines without sorting, preserving order: the first time a line is seen, seen[$0] is 0, so !0 is true and the line prints; the ++ then makes it 1 so later copies are skipped. This is one of the best-known awk one-liners.

Built-in functions

Function Result
length(s) length of a string (length alone = length of $0)
substr(s, start, len) substring, 1-based
index(s, t) position of t in s, or 0
split(s, arr, sep) split into an array, returns the count
sub(re, repl, target) / gsub(...) replace the first / all matches (default target $0)
match(s, re) position of the match; sets RSTART and RLENGTH
toupper(s) / tolower(s) change case
sprintf(fmt, ...) format into a string
int(x), sqrt(x), exp, log, sin, cos, rand() math
Bash String and math functions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Careful with sub and gsub: the pattern is a regex (/-/ or "-"), and in the replacement & means the matched text, so write \\& for a literal ampersand. POSIX awk regexes are EREs (see Regular Expressions) without lookahead. In mawk 1.3.3 (our sandbox), interval expressions like /a{3}/ are not supported (the pattern silently matches nothing useful), while gawk and newer mawk versions do support them; for portable scripts write [0-9][0-9][0-9] instead of [0-9]{3}. Also note that printf "%.0f" rounds 2.5 to 2 and 3.5 to 4 (round half to even, as in C). Other gawk-only features you will meet online: gensub, strftime, systime, asort, length(array) (an error in our mawk), PROCINFO. Check awk -W version (mawk) or awk --version (gawk) when something fails.

Putting it together: a report

Formatted reports combine a BEGIN header, per-line accumulation, and an END summary with printf. Let's build a small sales-by-region report from CSV-like data.

Bash A formatted report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Two details: printf ... | "sort" sends the lines to the sort command so the regions come out alphabetically (awk starts the command and you close() it to flush before printing the total), and the multi-line program shows awk's layout: newlines separate statements, comments start with #.

awk programs in files

When a program grows past a few lines, save it in a file and run it with -f. A script can start with a shebang to be executable (#!/usr/bin/awk -f), but note that the path of awk differs between systems. Keeping the program in a file avoids all shell-quoting problems.

Bash awk -f with a program file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

When to use awk (and when not)

Use awk when the job is "for each line, look at columns, maybe aggregate". Use grep for plain filtering, sed for text substitution, sort, uniq, cut and wc for simple counting, and a scripting language (Python, Perl) when you need real data structures, JSON, dates or libraries. A good rule: if the awk program no longer fits on one screen, it probably deserves a proper script. Typical awk pitfalls are worth a last look:

  • Quotes: the program is in single quotes, so it cannot contain a single quote. Use \047 or -f file.
  • print a b vs print a, b: juxtaposition concatenates; the comma inserts OFS.
  • Numeric strings: "10" < "9" is true as text. Convert with + 0 when you want numbers.
  • Uninitialised variables are 0/empty, which hides typos: totl += $1 silently makes a new variable.
  • Fields count from 1. $0 is the whole line, and $(NF-1) is the second-to-last field (use parentheses).
  • CSV: -F, breaks on quoted commas.
  • Locale: string comparison and sort can differ by locale; this course sets LC_ALL=C.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does awk '{ print $NF }' file print?

    NF is the number of fields, so $NF is the last field. The count itself is NF, and $1 is the first field.
  2. 2. What is the difference between print $1, $2 and print $1 $2?

    A comma separates print items with OFS, which defaults to a space. Without it the two values are concatenated, for example ab instead of a b.
  3. 3. Which command sums the third comma-separated column of data.csv, skipping a header line?

    -F, sets the comma separator and NR > 1 skips the header. Without -F the whole line is field 1, so $3 would be empty, and the last command only prints a field of the final line.
  4. 4. What does awk '!seen[$0]++' file do?

    seen[$0] starts at 0, so the first time a line appears the pattern is true; ++ then marks it as seen so duplicates are skipped.
  5. 5. Why should you pass a shell variable to awk with -v limit="$n" instead of writing awk "$2 > $n"?

    In double quotes the shell expands $2 before awk runs. -v passes the value as data, so awk's own program stays in single quotes.

Summary

  • An awk program is pattern { action } pairs; no pattern means every line, no action means print.
  • Fields: $1, $2, $NF, $0; NR is the line number, NF the field count; -F sets the input separator (a character or regex), OFS the output separator.
  • Patterns are regexes, comparisons, line numbers, ranges, BEGIN and END; combine them with &&, || and !.
  • Variables need no declaration; END is the place for totals; pass shell values with -v.
  • Associative arrays give counting and grouping: count[$1]++, sum[$1] += $2, for (k in arr) (pipe through sort), and !seen[$0]++ removes duplicates.
  • printf, length, substr, split, sub, gsub, match, toupper and friends make reports and clean-up easy; check which awk you have before using gawk-only features.
  • Keep long programs in a file and run them with awk -f.

Next: sort, uniq, cut, tr and wc