Skip to content
Tutorials Center
My learning

Topics

Lesson 67 of 95 ยท Projects

Project: Log Analyzer

Build a log analyzer in Bash: generate a sample access log, then use awk, sort and uniq to report top IPs, status codes, hourly traffic, errors and bandwidth.

Every server writes logs, and when something is wrong, the logs are where the answer lives. Opening a file with 500,000 lines in an editor is not an analysis; a few well-chosen shell commands are. In this project you build a log analyzer: first by exploring a web server access log one question at a time, and then by packaging the answers into a reusable script with a clean report.

You need what you learned in awk, sort, uniq, cut, tr and wc and Building Text Pipelines. The skeleton of the final script follows the habits from Designing Maintainable Scripts. Because the sandbox has no real server, the first step generates a realistic, fully deterministic sample log.

The log format

Web servers such as nginx and Apache write the Common Log Format, one request per line:

TEXT
203.0.113.7 - - [05/Jan/2026:10:15:32 +0000] "GET /index.html HTTP/1.1" 200 5120

Split on whitespace, the fields that matter are:

Field Meaning Example
$1 Client IP address 203.0.113.7
$4 Time, starting with [ [05/Jan/2026:10:15:32
$6 Method, starting with " "GET
$7 Request path /index.html
$9 HTTP status code 200
$10 Bytes sent 5120

Because the time field contains no spaces, awk's default field splitting is enough. (Fields that can contain spaces, such as the user agent in the combined format, are quoted and need more care.)

Step 1: generate a sample log

To keep results identical on every run, the generator uses its own tiny pseudo-random function instead of $RANDOM: a linear congruential generator (a formula that turns each number into the next one) with a fixed start. The same start always produces the same log.

Bash Step 1: generate 400 deterministic log lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Generating data with awk is a valuable trick in its own right: you now have a repeatable test fixture. Real logs are also easy to simulate by hand: write a dozen lines in a here-doc, including bad ones, and test your script on that before pointing it at production data.

Step 2: who and what, with sort and uniq

The classic question is "what is the most frequent value in this column?". The answer is always the same four-stage pipeline: extract, sort, count, sort again.

TEXT
  access.log --awk '{print $1}'--> ips --sort--> grouped --uniq -c--> counts --sort -rn--> ranked

uniq only collapses adjacent identical lines, which is why the first sort is not optional.

Bash Step 2: top IPs, top paths and status codes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

One important detail: ties. Two paths with the same count appear in whatever order sort -rn leaves them. For a stable, reproducible report, add a second sort key, for example sort -k1,1nr -k2 (count descending, then name ascending). The final script does exactly that.

You can also do the counting inside awk with an associative array. It is usually faster on big files because it needs one pass and no intermediate sorting of 500,000 lines, and it lets you compute several things at once:

Bash Step 2b: counting inside awk with arrays

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Step 3: traffic over time

Time buckets answer "when are we busy?". The hour sits inside field 4, [05/Jan/2026:10:15:32. Splitting that field on : gives the hour as its second piece. awk's split(string, array, separator) does it; note that mawk (the awk in our sandbox) and gawk both support this.

A text histogram is a nice touch: print one # for every five requests.

Bash Step 3: requests per hour with a text histogram

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Two things to remember about time analysis. First, the timestamp text 05/Jan/2026 does not sort alphabetically (April would come before January), so for multi-day logs convert it, or sort on the numeric parts you extract. Second, logs have a time zone: group by the zone the server logs in, and convert only at the end.

Step 4: errors and bandwidth

Status codes are grouped by their first digit: 2xx success, 3xx redirection, 4xx client error, 5xx server error. A useful summary is the error rate and the paths that cause server errors. substr($9, 1, 1) extracts the first digit. For bandwidth, sum field 10 and convert to a friendly unit. A dash - in the bytes field (the server logs it when no body was sent) must be treated as 0.

Bash Step 4: error classes, failing paths and total bandwidth

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The test $9 >= 400 compares numerically because the field looks like a number. If you want to be strict, add +0 ($9 + 0 >= 400) to force a number. Notice also that awk conditions without an action print the matching line, which is why $9 ~ /^5/ { print $7 } needs an explicit action to print only the path.

Step 5: the finished tool

Now put it together as a script that takes a log file (or reads standard input), accepts -n COUNT for the length of the top lists, tolerates garbage lines, and prints one clear report. Two design points: all statistics are computed by one awk pass, which is efficient and keeps the counts consistent; and the sorting for display happens afterward, with explicit tie-breaks so the report is the same on every run.

Bash Step 5: analyze.sh, a complete log report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The report shows everything the single awk pass collected. A few points about the code are worth reading twice:

  • The first awk rule (NF >= 10 && $9 ~ /^[0-9][0-9][0-9]$/) defines what a valid line is, and the next statement stops further processing for it. Every other line, including the blank one, lands in the fallback rule that counts bad. A parser that never reports how many lines it skipped hides problems.
  • awk writes each table to a file in a private mktemp -d directory, removed by the EXIT trap. The regex uses [0-9][0-9][0-9] instead of [0-9]{3}, because the mawk in many distributions does not support intervals.
  • top() sorts by count (descending, numeric), then by key, so ties never change between runs.
  • INPUT=${1:--} makes - (standard input) the default, the Unix convention for "read the pipe".

Where to go next

Real-life extensions of this analyzer:

  • Parse the combined format (referrer and user agent are quoted strings). Use -F'"' to split on quotes, then split the first piece on spaces.
  • Follow a live log with tail -F access.log | analyze.sh, or count errors per minute and alert above a threshold.
  • Look at slow requests if your format logs the response time: sort on that column and print the top ten.
  • Detect abuse: IPs with hundreds of 404s or 401s in a minute are scanners or password guessers.
  • Rotated logs: zcat access.log.*.gz | analyze.sh analyzes compressed archives without unpacking them.
  • Beyond a few gigabytes, or for ad hoc questions across many servers, a log platform (or loading the data into a database) beats shell pipelines. For anything up to that point, awk is hard to beat.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why must there be a sort before uniq -c in awk '{print $1}' log | sort | uniq -c?

    uniq compares each line with the previous one. Without sorting, the same IP appearing in different places would be counted separately.
  2. 2. In the Common Log Format line 1.2.3.4 - - [05/Jan/2026:10:15:32 +0000] "GET /a HTTP/1.1" 200 512, which awk field holds the status code?

    Fields: $1 IP, $2 and $3 dashes, $4 [time, $5 +0000], $6 "GET, $7 path, $8 HTTP/1.1", $9 status, $10 bytes.
  3. 3. How do you get the hour out of the field [05/Jan/2026:10:15:32 in awk?

    Splitting on the colon gives t[1]="[05/Jan/2026", t[2]="10", t[3]="15" and t[4]="32".
  4. 4. Why does the final report sort with -k1,1nr -k2,2 instead of just sort -rn?

    With only a count key, rows with equal counts come out in an unspecified order. Adding the name as a second key makes the report stable, which matters for tests and for diffing two reports.
  5. 5. What does the pattern-only awk program awk '$9 >= 400' access.log print?

    A pattern without an action runs the default action, which is print $0. To print only part of the line you write an explicit action such as { print $7 }.

Summary

  • Common Log Format fields split on whitespace: IP $1, time $4, path $7, status $9, bytes $10.
  • Ranking pipeline: extract, sort, uniq -c, sort -rn; add a second sort key for stable ties.
  • awk arrays count, sum and group in one pass; the order of for (k in array) is unspecified, so sort afterward.
  • Split the timestamp on : for hourly buckets; classify statuses with substr($9, 1, 1) and treat - bytes as zero.
  • Validate input, count the lines you skip, and let the script read from a file or standard input.
  • Generate deterministic sample logs with an awk formula so that tests and examples are reproducible.

Next: Project: File Organizer