Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 38 ยท AWK Programs

AWK Associative Arrays: Counting, Grouping and Summing

Use AWK associative arrays to count words, group rows by key, compute totals and averages, test membership, delete entries and loop in a stable order.

Nearly every useful AWK program of more than one line uses an array. AWK arrays are associative: the index is a string, not a position, so count["apple"] is as natural as count[3]. That one idea turns a handful of lines into a word counter, a group-by report, a duplicate remover or a lookup table.

This lesson assumes you know fields, patterns and BEGIN/END from the earlier chapters. Everything runs in the editors below. Our sandbox awk is a small POSIX-style implementation, so the examples avoid features that only some versions have (like length(array) or asort); where that matters the text says so.

How an associative array behaves

You never declare an array. The first time you write n["x"]++, AWK creates the array and the element, starting from an empty value that counts as 0 in numbers and "" in strings.

AWK
count[$1]++          # one counter per distinct first field
total[$1] += $2      # one running total per key
seen[$0] = 1         # remember that this line occurred

Reading a missing element creates it (with an empty value). That is a classic trap: if (arr[k] == "") adds k to the array. To test for existence without creating it, use in:

AWK
if (k in arr) { ... }
if (!(k in arr)) { ... }

Two more facts: numeric subscripts are converted to strings (a[1] and a["1"] are the same element, but a[01] is a["1"] while a["01"] is different), and a comma subscript a[i, j] joins the parts with the special variable SUBSEP so you can fake a two-dimensional table.

Counting things

The famous idiom is a counter keyed by the thing you are counting. Loop with for (key in array) inside END.

Bash Word and status counters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The order of for (key in array) is unspecified: it can differ between AWK versions and even between runs. When the order matters, pipe the output to sort, as above. Gawk adds asort and PROCINFO["sorted_in"], but those are not portable.

Grouping and aggregating

Use the same key for several arrays to compute count, total and average at once.

Bash Group by department

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice !($2 in max): on the first row of a department the maximum is not set yet, so the in test avoids comparing against a created-empty value.

Removing duplicates

seen[$0]++ is true the second time a line appears, because the post-increment returns the old value (0, false, on first sight). Negating it gives a one-line uniq that needs no sorting and keeps the original order.

Bash Dedupe without sorting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Multi-dimensional tables with SUBSEP

a[i, j] really uses the single key i SUBSEP j. Test membership with (i, j) in a, and split a key back apart with split(key, parts, SUBSEP).

Bash A pivot table by region and month

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The statements r[$1] and m[$2] only touch the arrays (creating the keys); they are handy when you later want to loop over every distinct region or month.

Deleting and clearing

delete arr[k] removes one element. To empty a whole array portably, use a loop or split("", arr):

AWK
for (k in arr) delete arr[k]
split("", arr)        # also empties arr

Emptying between records is how you handle blocks: fill an array while reading one group, print a summary when the group ends, then clear it.

Counting elements

There is no portable length(arr). Keep your own counter, increment it when you add a new key:

AWK
if (!(k in arr)) { arr[k] = 1; n++ }

Arrays at a glance

AWK arrays in four ideas

String keys Any string or number is a key
Auto-create n[k]++ starts at zero
Test with in Reading creates, in does not
Order unknown Pipe to sort

Count, group, dedupe and join all start here.

1 / 2

Common mistakes

  • Relying on the order of for (k in a). Sort the output.
  • Testing if (a[k]) for existence: it creates a[k]. Use k in a.
  • Forgetting that fields are strings: max[$2] < $3 compares numerically only when both look numeric. Add +0 ($3 + 0) if in doubt.
  • Using length(a) on old AWKs: keep a counter instead.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does awk '!seen[$0]++' file do?

    seen[$0]++ returns the old count: 0 (false) the first time, so the negation is true only for first occurrences.
  2. 2. Why is if (arr[k] == "") a risky existence test?

    Referencing a missing element creates it with an empty value. Use k in arr to test without side effects.
  3. 3. How do you get a predictable output order from for (k in a)?

    The iteration order is unspecified, so sort the output (or iterate over a counter) when order matters.
  4. 4. What key does t["north", "jan"] use?

    AWK joins the parts with SUBSEP, so a comma subscript is still a single string key.

Summary

  • AWK arrays are associative: keys are strings, elements appear on first use.
  • count[key]++ and sum[key] += value are the building blocks of group-by reports.
  • k in arr tests membership without creating the element.
  • !seen[$0]++ removes duplicates while keeping order.
  • a[i, j] uses SUBSEP; delete a[k] or split("", a) clears data.
  • Iteration order is undefined: pipe to sort.

Next: String Functions and Regular Expressions in AWK