Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 38 ยท Real-world Tasks

Analyzing Web Server Access Logs with AWK and sed

Parse Apache and nginx access logs: count hits, top URLs and IPs, status code breakdown, requests per hour, bandwidth totals, error hunting and response times.

Log files are where AWK earned its reputation. A busy web server writes thousands of lines an hour, every line has the same shape, and the questions are always the same: Who is hitting us? Which pages are popular? How many errors? When is traffic highest? How much bandwidth did we use? One-line answers to these save you from installing an analytics package, and they work on any server you can log in to.

This lesson parses the combined log format used by Apache and nginx. All data in the examples is made up. If your logs differ, adapt the field numbers.

Anatomy of a log line

TEXT
203.0.113.7 - - [17/Mar/2025:08:45:10 +0000] "GET /index.html HTTP/1.1" 200 5123 "https://example.org/" "Mozilla/5.0"

With AWK's default whitespace splitting the fields are:

Field Value Meaning
$1 203.0.113.7 client IP
$4 [17/Mar/2025:08:45:10 time (with a leading bracket)
$6 "GET method (with a leading quote)
$7 /index.html requested path
$9 200 status code
$10 5123 response size in bytes

The referrer and user agent are quoted strings containing spaces, so fields after $10 are unreliable with default splitting. For those, split on the double quote: -F'"' makes $2 the request, $4 the referrer and $6 the user agent.

Bash A sample log, and the basics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Errors and suspicious activity

Status codes beginning with 4 are client errors and 5 are server errors. A first-digit test with substr catches a whole class at once.

Bash Finding problems

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Traffic over time

The timestamp [17/Mar/2025:08:45:10 hides the hour in characters 14 and 15 of the field (after removing the [). substr($4, 14, 2) is the hour, substr($4, 2, 11) the date.

Bash Requests per hour, with a bar chart

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Notice that hour 10 is missing from the output, because nothing happened then. If you want empty hours shown, loop over 00 to 23 yourself with sprintf("%02d", h) instead of looping over the array.

Bandwidth and response size

Field $10 is the size of the response body in bytes (- when none, which AWK reads as 0).

Bash Bytes served

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Splitting on quotes: user agents and referrers

With -F'"' the line breaks into quoted pieces: $1 is everything before the request, $2 the request line, $3 status and size, $4 referrer, $6 user agent.

Bash User agents and bots

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Putting it into a reusable script

Wrap the common questions in a script that takes the log file as an argument and prints a one-screen summary.

Bash A one-screen log summary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

(The length_of helper counts array elements portably, because length(array) is not available everywhere.)

Tips for real logs

  • Big files: AWK streams, so a multi-gigabyte log is fine. Put the cheapest filter first (grep before awk) to cut the work.
  • Rotated logs: zcat access.log.*.gz | awk ... analyses compressed history without unpacking.
  • A different format (nginx custom log_format, JSON logs) changes the field numbers: print NF and a few $n first to check your mapping.
  • Log data is untrusted input. Never run it through eval or paste it into a shell command.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a combined-format log line split on whitespace, which field is the HTTP status code?

    $7 is the path, $8 the protocol (HTTP/1.1"), $9 the status code and $10 the response size.
  2. 2. Why use -F'"' to read the user agent?

    Splitting on the quote character puts the quoted request, referrer and user agent each into a single field.
  3. 3. How do you extract the hour from [17/Mar/2025:08:45:10?

    After the opening bracket, the date takes 11 characters and the colon follows, so the hour starts at position 14.
  4. 4. What is a sensible first step for a log in an unfamiliar format?

    Always verify how fields map before computing anything from them.

Summary

  • Default whitespace splitting gives IP $1, time $4, path $7, status $9, bytes $10; use -F'"' for the quoted parts.
  • Counting with count[$field]++ gives top pages, IPs, status codes and bot share.
  • substr($4, 14, 2) extracts the hour for traffic charts.
  • Sum $10 for bandwidth, after checking it is numeric.
  • Put recurring questions into a script, and use zcat for rotated logs.

Next: CSV Reports and Data Cleaning with AWK