Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 38 ยท AWK Basics

AWK Fields and Separators: FS, OFS, RS and ORS

Control how AWK splits and joins text: the field separator FS and -F, regex separators, OFS, modifying fields and $0, NF arithmetic, and the record separators RS and ORS.

AWK's power comes from fields. Real data is separated by spaces, tabs, commas, colons, pipes, or sometimes by messy mixtures. In this lesson you will learn how to tell AWK where fields start and end (the input separators FS and RS), how to control the printed form (the output separators OFS and ORS), and how changing a field or NF rewrites the whole line.

The default: blanks

With no setting, FS is a single space, which is a special case: fields are split on runs of spaces and tabs, and leading and trailing blanks are ignored. That is why " a b " has two fields.

Setting FS: -F and BEGIN

You can set the field separator with the -F option or by assigning FS in BEGIN:

TEXT
awk -F: '{ print $1 }' /etc/passwd
awk 'BEGIN { FS = ":" } { print $1 }' /etc/passwd

Both do the same. A single character other than space is used literally, except that FS = "|" works as a plain pipe. A longer string is treated as a regular expression.

Bash Colon and comma separated data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Simple -F, splitting is not a real CSV parser: it breaks on quoted fields such as "Smith, John",42. The CSV recipes later in this course show how to handle that.

Regular-expression separators

When FS has more than one character it is a regex. This handles messy input, for example "a comma followed by optional spaces", or several different separators:

Bash Regex field separators

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The last two examples show the difference between a single-character separator (each | splits, so empty fields appear: 4 fields) and a regex ([|]+ means one or more pipes: 3 fields). Be careful to quote the regex so the shell leaves it alone. To use a tab: -F'\t'. If you need a literal separator that is also a regex metacharacter (.), use a class: -F'[.]'.

Fields can be assigned

Fields are variables you can change. When you assign to a field, AWK rebuilds $0 by joining all fields with the output field separator OFS (a space by default).

Bash Changing a field rebuilds $0

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The second and third commands show the classic idiom $1 = $1: assigning a field to itself forces AWK to rebuild the line with single OFS separators, which squeezes runs of blanks. The fourth converts spaces to dashes. This is a quick way to change the separator of a file (awk 'BEGIN{FS=","; OFS="\t"} {$1=$1; print}' converts CSV to TSV).

OFS and the comma in print

In print a, b the comma inserts OFS between the items. Without the comma, items are concatenated with nothing between them:

Bash Comma versus no comma

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

NF is writable

NF is the number of fields, and you can change it. Reducing NF drops the last fields; increasing it adds empty ones. Combined with $NF and arithmetic, you can address fields relative to the end:

Bash NF tricks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

$(NF-1) is the second-to-last field (parentheses are required: $NF-1 means "last field minus one"). NF = 3 truncates the record (a widely supported POSIX behaviour). The last command prints the fields in reverse order with a loop. Assigning to $(NF+2) adds empty field NF+1 and sets NF+2.

Splitting lines differently: RS and ORS

By default a record is a line: RS = "\n". Setting RS changes where records end:

  • RS = "" (empty): records are separated by one or more blank lines (paragraph mode); newline then also separates fields, in addition to FS.
  • RS = ";" or any single character: records end at that character.
  • RS as a regex (more than one character) is a gawk and mawk feature, not POSIX.

ORS is the output record separator, a newline by default, added by print.

Bash Paragraph mode and ORS

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

In paragraph mode, FS = "\n" makes each line a field. The second command prints the first field of each non-empty line (blank lines have no fields, so $1 is empty and AWK prints an empty item between separators). The third shows a custom record separator; the last record has no trailing ;.

Whitespace in the data

When fields contain spaces (Ada Lovelace), the default FS splits them. Use another separator (-F:), or prepare the data first. Another frequent surprise: a line ending in \r (Windows) leaves a carriage return at the end of the last field. Strip it with sub(/\r$/, "") at the start of the action.

Bash Windows line endings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How many fields does " a b " have with the default FS?

    The default separator splits on runs of blanks and ignores leading and trailing ones, giving the fields a and b.
  2. 2. What does echo 'a b' | awk '{ $1 = $1; print }' print?

    Assigning a field rebuilds $0 with OFS (a single space), so the extra blanks are squeezed.
  3. 3. What is $(NF-1)?

    The parentheses make AWK compute NF-1 first and then take that field; $NF-1 would subtract one from the last field's value.
  4. 4. How do you set the field separator to a tab on the command line?

    -F takes the separator; \t is a tab in AWK's string syntax.
  5. 5. What does RS = "" do?

    An empty RS means records are paragraphs separated by one or more blank lines, and newline becomes an extra field separator.

Summary

  • FS (or -F) says how input records are split into fields; a longer string is a regex; a single space is the default blank-splitting mode.
  • OFS is inserted between fields by print a, b and when $0 is rebuilt after a field changes; $1 = $1 forces a rebuild.
  • NF is writable; $(NF-1) addresses fields from the end.
  • RS and ORS control record separation on input and output; RS = "" is paragraph mode.
  • Handle \r from Windows files and don't mistake -F, for a CSV parser.

Next: Patterns and Ranges