Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 43 ยท Projects

Project: CSV Report Generator

Build a Perl report generator: read a CSV with Text::CSV, validate rows, group and total sales by region, print an aligned table and handle bad data.

Almost every company has data in CSV files, and almost every manager wants it as a tidy report. Turning one into the other is a classic Perl job: read rows, check them, group and add up numbers, and print an aligned table. In this project you will build that tool step by step.

You will use ideas from earlier lessons: hashes, file input and output and regular expressions. The only new piece is the CPAN module Text::CSV, which reads CSV correctly.

Perl is not installed on this site, so the Perl code is read-only. Its output below was captured from a real run (Perl 5.42). Text::CSV is not a core module, so for that run a tiny stand-in module that splits on commas was used; the report logic is the real code. The interactive examples are Python stand-ins that do the same job with Python's csv module.

The goal

The input file sales.csv has one sale per line:

TEXT
region,product,units,price
North,Widget,10,2.50
South,Widget,4,2.50
North,Gadget,3,10.00
East,Gadget,x,10.00
South,Gizmo,7,4.25
North,Gizmo,2,4.25

Notice that one row is broken: East has x where the number of units should be. A good tool must not crash on it and must not silently count it. It should skip the row and tell you which line was wrong.

The report pipeline

Read Parse each CSV row into a hash.
Validate Reject rows with bad numbers.
Group Add units and revenue per region.
Print Sort and format an aligned table.

Each step is a few lines of Perl.

1 / 2

Why not just split on commas?

The quick way, split /,/, $line, breaks as soon as a value contains a comma inside quotes, such as "Smith, Jo", or a quote character, or a line break inside a field. CSV has rules for all of these, and Text::CSV follows them. Install it with cpanm Text::CSV (see CPAN); Text::CSV_XS is installed too if available and makes it faster.

Try the problem in the editor. The Python csv module plays the role of Text::CSV, and a plain split shows what goes wrong.

Python Split versus a real CSV parser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The split cuts Widget, large into two fields and shifts every column after it. A real parser keeps it as one field.

Reading rows as hashes

Text::CSV reads one row at a time with getline, which returns an array reference. After you set the column names from the header row, getline_hr returns a hash reference instead, so you can write $row->{units} and not remember that units is the third column.

PERL
use v5.36;
use Text::CSV;

my $file = shift // 'sales.csv';

my $csv = Text::CSV->new({ binary => 1, auto_diag => 1 });
open my $fh, '<:encoding(UTF-8)', $file or die "Cannot open $file: $!";

$csv->column_names($csv->getline($fh));

Three details are worth knowing:

  • binary => 1 allows any characters, including UTF-8 text and embedded newlines. Always set it.
  • auto_diag => 1 makes the module report malformed CSV (for example an unclosed quote) with a clear message instead of returning nothing.
  • The file is opened with the :encoding(UTF-8) layer so that text is decoded properly. A failed open calls die with $!, the operating system's reason.

Validating and grouping

Now loop over the rows. For each one, check that units is a whole number and price is a decimal number. A row that fails goes to a @rejected list with its line number and is skipped with next. A good row adds into two hashes keyed by region.

PERL
my (%revenue, %units, @rejected);
my $line = 1;    # the header was line 1
while (my $row = $csv->getline_hr($fh)) {
    $line++;
    my ($region, $n, $price) = @{$row}{qw(region units price)};

    unless (defined $n && $n =~ /\A\d+\z/ && defined $price && $price =~ /\A\d+(?:\.\d+)?\z/) {
        push @rejected, "line $line: bad number in "
            . join(',', map { $_ // '' } @{$row}{qw(region product units price)});
        next;
    }
    $units{$region}   += $n;
    $revenue{$region} += $n * $price;
}
close $fh;

Look at how the pieces fit:

  • @{$row}{qw(region units price)} is a hash slice through a reference. It fetches three values at once.
  • \A and \z anchor the pattern to the start and end of the whole string, so 10abc fails. Plain ^ and $ would also allow a trailing newline.
  • += on a missing hash key starts from zero, with no warning. This is the same trick as in hash patterns.
  • $_ // '' replaces undef with an empty string, so the message never triggers a "use of uninitialized value" warning.

Printing the report

Sort the regions by revenue, highest first, and use printf to line up the columns. Format codes control the width: %-8s is a left-aligned string in 8 characters, %6d an integer in 6, and %10.2f a number with two decimals in 10.

PERL
my $total = 0;
printf "%-8s %6s %10s\n", 'Region', 'Units', 'Revenue';
printf "%-8s %6s %10s\n", '-' x 8, '-' x 6, '-' x 10;
for my $region (sort { $revenue{$b} <=> $revenue{$a} } keys %revenue) {
    printf "%-8s %6d %10.2f\n", $region, $units{$region}, $revenue{$region};
    $total += $revenue{$region};
}
printf "%-8s %6s %10.2f\n", 'TOTAL', '', $total;

if (@rejected) {
    print STDERR "Skipped rows:\n";
    print STDERR "  $_\n" for @rejected;
}

Running perl report.pl sales.csv printed this table, and the warning about the broken row went to standard error:

TEXT
Region    Units    Revenue
-------- ------ ----------
North        15      63.50
South        11      39.75
TOTAL               103.25
TEXT
Skipped rows:
  line 5: bad number in East,Gadget,x,10.00

Sending problems to STDERR and the report to STDOUT is a good habit. You can redirect the report into a file with perl report.pl sales.csv > report.txt and still see the warnings on screen.

The x operator in '-' x 8 repeats a string, which is why the divider line is built without typing the dashes.

The same logic in a runnable stand-in

Here is the whole pipeline in Python. The structure matches the Perl code line by line: a reader that yields dictionaries, a validation step, two grouping dictionaries and a formatted table. Change a number in the data and press Run to watch the row get rejected.

Python CSV report pipeline (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ideas to extend it

  • Add a --min-units option with Getopt::Long (the next project, the batch file renamer, uses it).
  • Group by two keys at once with a hash of hashes: $sales{$region}{$product} += $n.
  • Write the report back as CSV with $csv->say($out_fh, \@fields) after calling Text::CSV->new({ binary => 1, eol => "\n" }).
  • Wrap the loop body in eval { ... } and count errors, so that one unexpected row never stops a nightly job.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is Text::CSV better than split /,/ for reading CSV files?

    CSV allows commas inside quoted values. A plain split cuts those values apart, while Text::CSV follows the CSV rules.
  2. 2. After $csv->column_names($csv->getline($fh)), what does getline_hr return for each row?

    column_names stores the header row, and getline_hr uses those names as hash keys, so you can write $row->{units}.
  3. 3. Why does the validation use \A\d+\z and not ^\d+$?

    In Perl, $ also matches before a final newline. \z matches only at the true end, which is stricter and safer for validation.
  4. 4. Why are skipped rows printed to STDERR and the table to STDOUT?

    Separate streams let a user save or pipe the report without mixing problem messages into it.

Summary

  • Read CSV with Text::CSV (binary => 1), never with a plain split.
  • Set column_names from the header and use getline_hr to get hash references.
  • Validate each row with anchored regexes and skip bad rows with next, remembering their line numbers.
  • Group with hashes and +=, sort keys by value, and print with printf widths.
  • Send the report to STDOUT and warnings to STDERR.

Next project: Batch File Renamer.