Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 38 ยท Text Processing Pipelines

Multi-line Records, Paragraph Mode and CSV in AWK

Control how AWK splits input: custom RS and FS, paragraph mode with RS empty, multi-line records, fixed-width data, and parsing quoted CSV fields correctly.

By default AWK treats a line as a record and whitespace as the field separator. Those are just defaults. Two built-in variables, RS (record separator) and FS (field separator), let the same program read address books where each person spans several lines, files with ; between records, tab-separated reports and CSV files with quoted commas.

This lesson shows how to change them, the special paragraph mode, and how to cope with CSV, which is harder than it looks.

FS in depth

FS can be set with -F or in BEGIN. Its value is interpreted three ways:

  • a single space (the default): split on runs of blanks and tabs, and ignore leading/trailing blanks;
  • any other single character: split on exactly that character (so -F, makes a,,b three fields, one empty);
  • longer strings: treated as a regular expression, e.g. -F'[,;]' or -F' *: *'.

-Ft or -F'\t' selects a tab (in POSIX AWK -Ft means tab). Changing FS takes effect on the next record, not the current one; to re-split the current line, assign $0 = $0 after setting FS.

Bash Different separators

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last example prints p:q:r first (the old FS split the first record) and then p after re-splitting with $0 = $0.

OFS and rebuilding the line

OFS is the string used to join fields when AWK rebuilds $0. Rebuilding happens whenever you assign to a field. A harmless trick, $1 = $1, forces the rebuild, which converts any input separator to OFS:

Bash Converting separators

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Changing RS: other record separators

Set RS to a single character to change what ends a record. This is how you read comma-separated lists, NUL-less records, or a stream where ; separates items. A few AWKs (gawk, mawk) also accept a regular expression in RS, but a single character is portable.

Bash Records separated by semicolons

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The final record still contains the trailing newline (blue\n), so the output shows an extra blank line. Strip it with gsub(/\n/, "") or by printf 'red;green;blue' | ....

Paragraph mode: RS = ""

Setting RS to the empty string switches to paragraph mode. Records are separated by one or more blank lines, and โ€” importantly โ€” newline becomes an additional field separator (on top of FS). That is perfect for address books, config blocks or Key: value listings.

Bash An address book in paragraph mode

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Setting FS = "\n" makes each line of the paragraph one field, so spaces inside a line (like Ann Lee) do not split it.

Multi-line records without blank lines

When records have a start marker rather than blank-line separators (a log entry starts with a timestamp, a section starts with [name]), use a state variable instead of RS:

Bash Grouping lines under section headers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Fixed-width fields

For mainframe-style reports where columns sit at fixed positions, skip FS and use substr (gawk also has FIELDWIDTHS, but it is not portable).

Bash Fixed-width columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

CSV: why -F, is not enough

Real CSV allows a comma inside quotes ("Smith, John",42), doubled quotes for a literal quote ("say ""hi""") and sometimes line breaks inside quotes. Splitting on every comma breaks those rows. Gawk 5.3+ has a --csv mode, but you cannot count on it. A portable approach is to write a small parser function that walks the line character by character.

Bash A quote-aware CSV splitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

This handles quotes, escaped quotes and empty fields. A quoted newline would need joining physical lines first (keep reading lines with getline until the quote count is even). For heavy CSV work consider a dedicated tool such as csvkit or mlr (Miller), and reserve AWK for simple, well-behaved files.

Common mistakes

  • Setting FS in the main rule and expecting it to change the current line: use BEGIN or $0 = $0.
  • Forgetting that paragraph mode also splits on newlines; set FS = "\n" when lines contain spaces.
  • Assuming -F, handles quoted CSV.
  • Using FS=" " expecting a single literal space: that is the default whitespace mode; use [ ] for a literal space.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does setting RS = "" do?

    An empty RS means paragraph mode, with newline added to the field separators.
  2. 2. When does a change to FS take effect?

    The current record has already been split. Use $0 = $0 to re-split it with the new FS.
  3. 3. Why is awk -F, '{print $2}' unreliable for CSV?

    A quoted field like "Smith, John" contains a comma that is not a separator.
  4. 4. What does $1 = $1 achieve?

    Assigning to any field rebuilds the record with OFS between fields.

Summary

  • FS can be a single space (default), one character, or a regex; changes apply to the next record.
  • OFS joins fields when AWK rebuilds a record; $1 = $1 triggers it.
  • RS sets the record separator; RS = "" is paragraph mode and adds newline to the field separators.
  • Use a state variable for section-style data, substr for fixed-width data.
  • Parse quoted CSV with a character-level function, not -F,.

Next: sed One-liners