Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 43 ยท Files and I/O

Text Processing Idioms

Write compact Perl text filters: the default variable $_, while(<>) loops, -n and -p switches, field splitting, formatted reports with printf, and here-docs.

Perl was built to turn text into other text, and over the years a handful of idioms have grown up around that job. They make scripts short and readable once you know them: the default variable $_, the while (<>) loop, split for fields, printf for aligned columns, and here-docs for multi-line text.

This lesson puts those idioms together into small, complete filters. You will build a parser for comma-separated data, a formatted report, and a template for a message. It builds on Reading and Writing Files and on the regex lessons, starting with Substitution and Transliteration.

Perl is not installed on this site, so the Perl code is read-only, with output shown below it. The interactive examples are Bash and awk stand-ins: awk is the classic Unix cousin of this style of Perl, and the same filters can be expressed in it almost line by line.

The default variable $_

Many Perl operations quietly work on $_ when you do not name a variable. The loops for, foreach and while (<>) set it, and these functions and operators read it: print, chomp, split, lc, uc, length, pattern matches and s///. Compare the two versions of the same loop:

PERL
for my $line (@lines) {
    chomp($line);
    print lc($line), "\n" if $line =~ /error/i;
}

for (@lines) {
    chomp;
    print lc, "\n" if /error/i;
}

The second one says the same thing with less ceremony. Because for aliases $_ to each array element, changes to $_ change the array itself, so s/^\s+// inside a for (@lines) loop trims the original lines.

Use the default variable for short, obvious loops. When a body gets longer than a few lines, or when you call a subroutine inside it, a named variable such as $line is easier to read and safer, because other code may change $_ underneath you.

while (<>) loops

The pattern from the file lesson is the heart of every filter. while (<>) reads one line at a time from the files named on the command line (or from standard input) and, as a special rule, assigns it to $_:

PERL
use v5.36;

while (<>) {
    chomp;
    next if /^#/ or /^\s*$/;      # skip comments and blank lines
    say "$.: $_";
}

This pattern needs a few conventions to stay tidy:

  • chomp; removes the newline from $_.
  • next if ... skips lines you do not want, so the rest of the body stays unindented.
  • $. is the current line number, and $ARGV the current file name.
  • In a self-contained script, <DATA> reads the lines after a __DATA__ marker at the end of the program itself. It is perfect for examples and small tests.

The -n and -p switches

For quick jobs you do not even need a script file. The -n switch wraps your code in while (<>) { ... }, and -p does the same but prints $_ after every pass:

Terminal
perl -ne 'print if /ERROR/' app.log       # like grep ERROR
perl -pe 's/colour/color/g' notes.txt     # edit every line, print the result
perl -lane 'print $F[0]' data.txt         # print the first word of each line

-l handles newlines for you, and -a splits each line into the array @F. These switches get a lesson of their own in Perl One-Liners.

The filter pattern

Read while (<>) puts each line in $_.
Clean chomp, skip comments and blanks.
Split Break the line into fields.
Report Add up, then print with printf.

Almost every text tool you write fits these four steps.

1 / 2

Splitting lines into fields

Most data lines are records with fields. split with a pattern turns a line into a list, and a list assignment gives the pieces names:

PERL
my ($item, $qty, $price) = split /,/;

Here the pattern /,/ is applied to $_. A few rules prevent classic mistakes:

  • For whitespace-separated words, use split ' '. It ignores leading spaces and treats runs of spaces and tabs as one separator.
  • split /,/ drops empty trailing fields. Pass -1 as a limit (split /,/, $_, -1) when empty last columns matter.
  • A pattern like /\|/ is needed for a literal pipe. Characters such as ., | and + are special in regexes.
  • split is for simple delimiter formats. Real CSV with quoted commas needs the Text::CSV module (see the CSV project later in this tutorial).

Here is a complete small filter. It reads comma-separated order lines from the end of the script, skips comments and blanks, and totals the money per item:

PERL
use v5.36;

my %total;
while (<DATA>) {
    chomp;
    next if /^#/ or /^\s*$/;
    my ($item, $qty, $price) = split /,/;
    $total{$item} += $qty * $price;
}

printf "%-8s %8.2f\n", $_, $total{$_} for sort keys %total;

__DATA__
# item,qty,price
apple,3,0.50
pear,2,0.75
apple,2,0.50
kiwi,10,0.20
TEXT
apple        2.50
kiwi         2.00
pear         1.50

The hash %total starts empty, and += on a missing key starts from zero, so no setup is needed. The awk version below does exactly the same:

Bash Parse, skip and total (awk stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Reports with printf and sprintf

printf formats values into aligned columns; sprintf does the same but returns the string instead of printing it. The most useful conversions:

Format Meaning
%s string
%d integer
%.2f floating-point number with 2 decimals
%5d integer right-aligned in 5 columns
%-8s string left-aligned in 8 columns
%05.1f zero-padded, 1 decimal, 5 columns wide
%% a literal percent sign

Text is normally left-aligned and numbers right-aligned, so use %-Ns for names and %Nd or %N.2f for numbers. Build rules and borders with the repeat operator x:

PERL
use v5.36;

my %qty   = (apple => 5, kiwi => 10, pear => 2);
my %total = (apple => 2.5, kiwi => 2, pear => 1.5);

printf "%-8s %5s %8s\n", "Item", "Qty", "Total";
say "-" x 23;

my ($sum_qty, $sum_total) = (0, 0);
for my $item (sort keys %qty) {
    printf "%-8s %5d %8.2f\n", $item, $qty{$item}, $total{$item};
    $sum_qty   += $qty{$item};
    $sum_total += $total{$item};
}
printf "%-8s %5d %8.2f\n", "TOTAL", $sum_qty, $sum_total;
TEXT
Item       Qty    Total
-----------------------
apple        5     2.50
kiwi        10     2.00
pear         2     1.50
TOTAL       17     6.00

When the column width is not known in advance, put a star in the format and pass the width as an argument: printf "%-*s|\n", $width, $text. Compute the longest name first with a loop, and your table will adapt to the data. Perl also has a built-in format/write report system, but it is dated, and printf covers nearly every case.

Bash A formatted report (awk stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Here-docs

A here-doc lets you write a block of text directly in the program, which is ideal for messages, SQL, usage help and templates. You choose a terminator word, put <<"WORD" after print, and the text runs until a line that contains only that word:

PERL
use v5.36;

my ($name, $count) = ("Ana", 3);

print <<"END";
Dear $name,

You have $count new messages.
END
TEXT
Dear Ana,

You have 3 new messages.

The quoting of the terminator decides what happens inside, just like for ordinary strings:

  • <<"END" (or plain <<END) interpolates variables and escapes such as \n.
  • <<'END' is literal: $name stays $name, so it suits code samples and regex-heavy text.
  • <<~END (Perl 5.26 and later) removes the common leading whitespace, so you can indent the here-doc inside a block and the closing END too:
PERL
sub usage {
    print <<~'END';
        Usage: report.pl [file ...]
          -h   show this help
        END
}

That prints two lines, with the indentation of the least-indented line removed and the extra two spaces before -h kept.

Combine here-docs with s///e (see Substitution and Transliteration) or with sprintf for small template systems, or with a hash of values interpolated as @{[ ... ]} for expressions. For anything bigger, use a template module from CPAN.

Bash Here-documents (Bash stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In while (<>) { chomp; print if /error/; }, which variable do chomp and the match work on?

    while (<>) assigns each line to $_, and chomp, print and a bare pattern match all default to $_.
  2. 2. What does split ' ', $line do differently from split / /, $line?

    A single-space string is a special case meaning awk-style splitting. A real pattern / / splits on every single space and keeps empty fields.
  3. 3. Which format prints 7 as 7, right-aligned in four columns?

    %4d pads on the left to a width of 4. %-4d pads on the right, %04d pads with zeros and %.4d sets a minimum number of digits.
  4. 4. What is the difference between <<"END" and <<'END'?

    The quotes around the terminator behave like ordinary string quotes, so <<"END" interpolates and <<'END' does not.
  5. 5. What does the -p switch do in perl -pe 's/a/b/' file?

    -p loops over every input line like -n, but also prints $_ at the end of each iteration, which makes it ideal for editing text.

Summary

  • Many Perl built-ins, loops and regex operations default to $_; use it in short loops and name your variables in longer ones.
  • while (<>) with chomp, next if and split is the standard shape of a text filter; <DATA> reads from the end of the script.
  • -n and -p turn code into line-by-line filters from the command line.
  • split ' ' handles whitespace-separated words; add a limit of -1 to keep empty trailing fields.
  • printf and sprintf build aligned reports with %-8s, %5d and %8.2f; x makes rules and borders.
  • Here-docs (<<"END", <<'END', <<~END) hold multi-line text, and the quotes decide whether variables are interpolated.

Next, learn how Perl stores nested data in References.