Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 43 ยท Projects

Project: Log Analyzer

Build a Perl log analyzer step by step: parse access log lines with a regex, count requests by status, IP and path, and print a top-N report.

Log files are where Perl feels most at home: thousands of lines of semi-structured text, and a few questions to answer. Which pages fail? Which clients send the most traffic? How many bytes did we serve? In this project you will build a small, tidy log analyzer that reads web-server access logs, counts requests in several ways, and prints a short report.

Along the way you will use nearly everything from the first half of the tutorial: a named-capture regex with /x, while (<>), hashes as counters, sorting by value, printf for aligned columns, and a subroutine with a signature. Each step is explained, then the full program is shown together.

Perl is not installed on this site, so the Perl code below is read-only; the report shown comes from a real run of the program on the sample log. The interactive example is a Python stand-in that does the same job with the same regular expression idea, so you can edit the log and the pattern and watch the report change.

The input: an access log

Most web servers can write the "common log format". Each request is one line:

TEXT
203.0.113.5 - - [12/Mar/2026:10:15:32 +0000] "GET /index.html HTTP/1.1" 200 5120

From left to right: the client IP address, two fields we ignore (identity and user name), the time in square brackets, the request in double quotes (method, path and protocol), the HTTP status code, and the number of bytes sent (or - when there is none). The addresses here come from the ranges reserved for documentation, so they are safe to use in examples.

Save the following as access.log. It includes one broken line on purpose, because real logs always have a few:

TEXT
203.0.113.5 - - [12/Mar/2026:10:15:32 +0000] "GET /index.html HTTP/1.1" 200 5120
203.0.113.5 - - [12/Mar/2026:10:15:33 +0000] "GET /style.css HTTP/1.1" 200 1024
198.51.100.7 - - [12/Mar/2026:10:16:01 +0000] "GET /missing.html HTTP/1.1" 404 312
203.0.113.9 - - [12/Mar/2026:10:16:45 +0000] "POST /login HTTP/1.1" 302 0
198.51.100.7 - - [12/Mar/2026:10:17:10 +0000] "GET /index.html HTTP/1.1" 200 5120
203.0.113.5 - - [12/Mar/2026:10:17:30 +0000] "GET /api/items HTTP/1.1" 500 87
not a valid log line
203.0.113.9 - - [12/Mar/2026:10:18:02 +0000] "GET /index.html HTTP/1.1" 200 5120
198.51.100.7 - - [12/Mar/2026:10:18:44 +0000] "GET /missing.html HTTP/1.1" 404 312

The plan

Read One line at a time with while (<>).
Parse A named-capture regex splits each line.
Count Hashes tally status, client and path.
Report Sort and print the top results.

Bad lines are counted and skipped, never fatal.

1 / 2

Step 1: a regex for one line

The pattern uses the /x modifier so we can lay it out over several lines and use spaces freely (see Captures and Modifiers). Each interesting part gets a name:

PERL
my $line_re = qr{
    ^ (?<ip>\S+) \s+ \S+ \s+ \S+ \s+
    \[ (?<time>[^\]]+) \] \s+
    " (?<method>[A-Z]+) \s+ (?<path>\S+) \s+ [^"]* " \s+
    (?<status>\d{3}) \s+ (?<bytes>\d+|-)
}x;

Read it piece by piece:

  • (?<ip>\S+) grabs the first run of non-space characters. The two \S+ that follow skip the identity and user fields.
  • \[ (?<time>[^\]]+) \] matches the square brackets literally (they are escaped) and captures everything up to the closing one. [^\]]+ means "one or more characters that are not a closing bracket".
  • The quoted request is split into the method ([A-Z]+), the path (\S+) and the protocol, which we skip with [^"]* (anything except a closing quote).
  • (?<status>\d{3}) is exactly three digits, and (?<bytes>\d+|-) is a number or a dash.

Because the pattern is stored with qr{} it is compiled once and reused for every line, which matters for big logs (see Performance and Profiling). Using {} as the delimiter lets the pattern contain / without escaping.

Step 2: read, match and count

The diamond operator <> reads the files named on the command line, or standard input when there are none. That makes the program work as perl logan.pl access.log and also as cat access.log | perl logan.pl.

PERL
my (%by_status, %by_ip, %by_path);
my ($total, $bad, $bytes) = (0, 0, 0);

while (my $line = <>) {
    chomp $line;
    if ($line !~ $line_re) {
        $bad++;
        next;
    }
    my %r = %+;
    $total++;
    $by_status{ $r{status} }++;
    $by_ip{ $r{ip} }++;
    $by_path{ $r{path} }++;
    $bytes += $r{bytes} eq '-' ? 0 : $r{bytes};
}

Key points:

  • A line that does not match is counted in $bad and skipped with next. The program never dies because of one broken line.
  • After a successful match, %+ holds the named captures. Copying it into %r straight away is wise, because the next match would replace it.
  • $by_status{ $r{status} }++ is the standard counting idiom: a missing key counts as zero, so the first ++ makes it one. See Hash Patterns.
  • The ternary expression treats a - byte count as zero so the addition stays numeric.

Step 3: the top-N helper

We need "the three most frequent keys" three times, so it deserves a subroutine. It sorts the keys by count, largest first, and breaks ties alphabetically so the report is the same every run. Hash order in Perl is deliberately unpredictable, so an explicit tie-breaker is important.

PERL
sub top_n ($counts, $n) {
    my @keys = sort { $counts->{$b} <=> $counts->{$a} || $a cmp $b } keys %$counts;
    $#keys = $n - 1 if @keys > $n;
    return @keys;
}

The sort block compares counts numerically with <=>, and only when they are equal does || fall through to the string comparison cmp. Setting $#keys shortens the array to the first $n elements. The subroutine takes a hash reference, as explained in References, so the call is top_n(\%by_ip, 3).

Step 4: print the report

PERL
say "Requests: $total   Skipped lines: $bad   Bytes sent: $bytes";
say "";
say "By status:";
printf "  %-4s %3d\n", $_, $by_status{$_} for sort keys %by_status;
say "";
say "Top clients:";
printf "  %-15s %3d\n", $_, $by_ip{$_} for top_n(\%by_ip, 3);
say "";
say "Top paths:";
printf "  %-15s %3d\n", $_, $by_path{$_} for top_n(\%by_path, 3);

printf formats are the same as in C: %-15s pads a string to 15 characters on the right, and %3d right-aligns an integer in three columns, which gives neat columns. The for statement modifier runs one printf per key.

The complete program

Put the pieces together, with use strict-style safety from the use v5.36 line (it enables strict, warnings, say and signatures):

PERL
#!/usr/bin/perl
use v5.36;

my $line_re = qr{
    ^ (?<ip>\S+) \s+ \S+ \s+ \S+ \s+
    \[ (?<time>[^\]]+) \] \s+
    " (?<method>[A-Z]+) \s+ (?<path>\S+) \s+ [^"]* " \s+
    (?<status>\d{3}) \s+ (?<bytes>\d+|-)
}x;

my (%by_status, %by_ip, %by_path);
my ($total, $bad, $bytes) = (0, 0, 0);

while (my $line = <>) {
    chomp $line;
    if ($line !~ $line_re) {
        $bad++;
        next;
    }
    my %r = %+;
    $total++;
    $by_status{ $r{status} }++;
    $by_ip{ $r{ip} }++;
    $by_path{ $r{path} }++;
    $bytes += $r{bytes} eq '-' ? 0 : $r{bytes};
}

sub top_n ($counts, $n) {
    my @keys = sort { $counts->{$b} <=> $counts->{$a} || $a cmp $b } keys %$counts;
    $#keys = $n - 1 if @keys > $n;
    return @keys;
}

say "Requests: $total   Skipped lines: $bad   Bytes sent: $bytes";
say "";
say "By status:";
printf "  %-4s %3d\n", $_, $by_status{$_} for sort keys %by_status;
say "";
say "Top clients:";
printf "  %-15s %3d\n", $_, $by_ip{$_} for top_n(\%by_ip, 3);
say "";
say "Top paths:";
printf "  %-15s %3d\n", $_, $by_path{$_} for top_n(\%by_path, 3);

Running perl logan.pl access.log prints:

TEXT
Requests: 8   Skipped lines: 1   Bytes sent: 17095

By status:
  200    4
  302    1
  404    2
  500    1

Top clients:
  198.51.100.7      3
  203.0.113.5       3
  203.0.113.9       2

Top paths:
  /index.html       3
  /missing.html     2
  /api/items        1

Check it against the log: eight valid requests, one skipped line, and 5120 + 1024 + 312 + 0 + 5120 + 87 + 5120 + 312 = 17095 bytes. The two clients with three requests each appear in address order because of the tie-breaker.

Try the same logic

This Python version follows the program step for step: the same regex (Python spells named groups (?P<name>...)), the same counters, and the same ordering rule. Change the log text, add a bad line, or ask for the top two instead of three:

Python Log analyzer (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ideas to extend it

  • Time: use the captured time to count requests per hour. Parse it with Time::Piece->strptime (see Dates and Times) or just capture the hour with another group.
  • Errors only: add a command-line option (with Getopt::Long) such as --status 5xx that prints only the matching lines.
  • Bandwidth: sum bytes per path and report the heaviest pages.
  • Bots: the combined log format adds referrer and user agent fields; extend the pattern to capture the user agent and count the most common ones.
  • Large files: the program already streams one line at a time, so memory use stays flat even for gigabyte logs. Compare run times on a large file with a profiler.
  • Tests: move the parsing into a module with a parse_line function and write Test::More tests for good and bad lines, as described in Testing with Test::More.

Treat log data as untrusted: it can contain anything an attacker typed into a URL. Never print it into HTML without escaping, and never pass a path or user agent to a shell. Security and Safe Perl explains why.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is the regex stored with qr{} and the /x modifier?

    qr// precompiles the pattern so it is reused for every line, and /x lets you spread the pattern over several lines so it stays readable.
  2. 2. Why does the program copy %+ into %r right after the match?

    Capture variables are overwritten by later matches, so you copy what you need into your own variables as soon as possible.
  3. 3. What does $by_ip{ $r{ip} }++ do the first time it sees a new IP address?

    A missing hash element counts as zero in numeric context, so incrementing it creates the key and makes it one.
  4. 4. Why does top_n break ties with $a cmp $b?

    When two keys have the same count, the order would depend on the hash. The string comparison makes the output the same on every run.

Summary

  • A named-capture regex (qr{...}x) turns each log line into named fields; non-matching lines are counted and skipped.
  • while (<>) streams input, so the program works on files or pipes with flat memory use.
  • Hashes with $h{$key}++ are the everyday counting tool; sort by count, then by key for stable output.
  • A small subroutine with a hash reference avoids repeating the top-N logic.
  • printf with widths produces readable columns.
  • The same structure extends easily to per-hour counts, filters, bandwidth totals and tests.

Next, build a second report tool in Project: CSV Report Generator.