Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 38 ยท Text Processing Pipelines

Working with Several Files and Joins in AWK

Read two or more files in one AWK run: FILENAME, FNR vs NR, the NR==FNR lookup idiom, joining and comparing files, getline, and passing values with -v.

Real data is rarely in one file. Employees are in one list and departments in another; yesterday's users and today's users; a blocklist and a log. SQL calls combining them a join. AWK can do it with a few lines and no database, as long as you understand how it numbers records across files.

This lesson covers the variables that distinguish "which file am I reading", the famous NR==FNR idiom for loading a lookup table, set operations on files, and the other ways to feed data to a program (-v, getline, ENVIRON).

NR, FNR and FILENAME

When you give AWK several files, it reads them one after another as a single stream.

  • NR counts records across all files (it never resets).
  • FNR counts records in the current file (it resets to 1 for each file).
  • FILENAME is the name of the file being read (empty in BEGIN; - for standard input).
Bash NR versus FNR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The two counters are equal only while reading the first file. That fact is the key to the next idiom.

The NR==FNR lookup idiom

To enrich rows of file B with data from file A, load A into an array while NR==FNR (we are still in the first file), then process B in the second rule. The next statement stops the first rule from falling through to the second.

AWK
NR == FNR { info[$1] = $2; next }     # file 1: store
{ print $0, info[$1] }                # file 2: use
Bash Join employees with departments

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The weakness: if the first file is empty, NR==FNR stays true for the second file too, so its rows are treated as lookup data. When empty files are possible, compare file names instead: FILENAME == ARGV[1] (or use FNR==1 { fileno++ }).

Set operations on files

Lists of ids, emails or paths are sets. Three one-liners cover most needs (assuming one item per line).

Bash Intersection, difference and union

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice how a[$0] alone (no assignment) creates the key, which is enough for a set, and how an expression with no action ($0 in a) prints matching lines.

Joining on several columns and merging fields

Use SUBSEP keys for composite joins, or append fields from file A to the end of each matching row in B.

Bash Add prices to order lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Comparing columns across two files line by line

Sometimes you want a side-by-side comparison, such as old and new values of the same key. Store file one by key, then print the differences.

Bash Report changed values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Passing values in: -v, ENVIRON, ARGV

Hard-coding values in the program is brittle. Three ways in:

  • awk -v limit=100 '$2 > limit' f assigns a variable before BEGIN runs. Backslash escapes in the value are processed.
  • ENVIRON["HOME"] reads environment variables, which avoids escape problems for arbitrary text: NAME=$x awk 'BEGIN { print ENVIRON["NAME"] }'.
  • A var=value argument placed between file names takes effect when AWK reaches it: awk '{...}' x=1 file1 x=2 file2.
Bash Getting values into a program

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Never build an AWK program by pasting untrusted text into the program string; pass it with -v or ENVIRON instead.

Reading another file or command with getline

getline reads a record on demand, from the current input or elsewhere. The three forms you need:

AWK
while ((getline line < "file") > 0) { ... }     # read a file line by line
"date +%Y" | getline year                       # run a command, read its output
close("file")                                   # reset so you can read again

Always test the return value: > 0 means a line was read, 0 is end of file, -1 is an error (for example a missing file). Writing while (getline line < "file") without the comparison loops forever on a missing file.

Bash getline from a file and a command

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Common mistakes

  • Using NR==FNR when the first file may be empty.
  • Forgetting next after loading the lookup table, so the second rule also runs on the first file.
  • Joining on keys with different whitespace or case: normalise with tolower and gsub first.
  • A getline loop without a > 0 test.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When reading two files, when are NR and FNR equal?

    NR keeps counting across files while FNR resets, so they match only in the first file.
  2. 2. What is the purpose of next in NR == FNR { a[$1] = $2; next }?

    next stops processing the current record, so the lookup rows are not also handled by the later rules.
  3. 3. Which command prints lines of new.txt that do NOT appear in old.txt?

    Load old.txt into a set, then print lines of the second file whose key is not in it.
  4. 4. How should you pass a user-supplied string into an AWK program safely?

    -v and ENVIRON pass data as values, not code, avoiding injection problems.

Summary

  • NR counts all records, FNR per file, FILENAME names the current file.
  • NR==FNR { store; next } loads a lookup table from the first file for the second.
  • Intersection, difference and union of files are one-liners using in and !seen[$0]++.
  • Pass data in with -v, ENVIRON, or var=value arguments.
  • getline reads on demand; always test its return value and close() when done.

Next: Combining sed and AWK in Scripts