Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 38 ยท Projects and Practice

Project: A Log Dashboard in One AWK Program

Build a complete AWK program that turns an access log into a text dashboard: summary, status table, top pages and visitors, hourly traffic chart and an error list.

Time to combine everything from the AWK chapters into one finished tool. The project is dashboard.awk: feed it a web server access log and it prints a one-screen text dashboard with totals, status codes, the busiest pages and visitors, a chart of traffic per hour, and a list of server errors. It uses arrays, functions, printf, -v options, BEGIN and END blocks, and a pipe to sort.

You will build it in four steps. Each step is complete and runnable, and the final listing is the whole program in one piece.

The design

flowchart TD
  A["access.log"] --> B["main rule<br/>count per record"]
  B --> C["arrays: ip, page, status, hour, bytes"]
  C --> D["END block"]
  D --> E["summary"]
  D --> F["status table"]
  D --> G["top pages / visitors<br/>(sorted)"]
  D --> H["hourly chart"]
  B --> I["server errors<br/>collected as lines"]
  I --> D

One pass over the file collects everything; the END block only formats.

The rule that keeps the program simple: the main block counts, the END block prints. Counting is one pass, memory use depends on the number of distinct IPs and pages, not on the size of the log.

Step 1: the data and the counting

First create a sample log of 12 lines and write the part that reads it.

Bash Step 1: count everything in one pass

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The statement errors[++nerr] = ... appends to a list: the counter nerr is incremented first, then used as the new index.

Step 2: helper functions

Small functions keep the END block readable: a bar of # characters, a count of distinct keys, and a "top N" printer that sorts through the shell.

Bash Step 2: helpers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

top() writes lines into a sort pipe and then close()s it, so the sorted lines appear before the next print. Without close() they would appear at the very end of the run. The fflush() after the title prints what AWK has buffered before sort starts writing, so each heading stays above its rows.

Step 3: the full program

Now put it in a file and run it. Note the variables you can set from the command line with -v (top and width).

Bash Step 3: dashboard.awk

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Look at how the program is organised: functions first, a BEGIN that sets defaults for top and width (so -v is optional), one counting block, and one reporting block. The exit inside END ends the program early for an empty log.

Step 4: try it, then extend it

Nothing in the editor is read-only: change the sample log, the -v values or the program and run it again. Suggested extensions, from easy to harder:

  1. Add a 404 list: collect the paths that returned 404 and print them under their own heading.
  2. Add a bot share: split the line on " and count user agents containing bot, curl or crawl.
  3. Add -v since=09 to skip entries before an hour.
  4. Print percentages beside the status codes.
  5. Make the chart scale to 100% width of a terminal using -v width=$(tput cols) (on a real terminal).
  6. Write the output as Markdown tables so it can be pasted into a README or issue.

What you practised

  • One pass, many arrays: the AWK way to aggregate.
  • Functions with local variables, array parameters and a sorted "top N" helper using a pipe and close().
  • Command-line options with -v and defaults in BEGIN.
  • Formatting with printf and text bars.
  • Keeping parsing (main rule) separate from reporting (END).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the dashboard separate a counting block from an END block?

    Totals are only complete after the last record, so the END block prints the report while the main block only collects data.
  2. 2. What does errors[++nerr] = ... do?

    The pre-increment gives the next free index, which builds a list.
  3. 3. Why call close(cmd) in the top-N helper?

    Closing the pipe makes sort finish and print before the next section of the report.
  4. 4. How do you let the user choose how many rows to show?

    -v assigns a variable before BEGIN, and a default can be set if it is empty.

Summary

  • A good AWK tool counts in the main block and reports in END.
  • Small helper functions (bar, distinct, show_top) keep the program readable.
  • -v with defaults in BEGIN makes programs configurable.
  • Pipes to sort plus close() give ordered sections.
  • The program is easy to extend because each section reads from the arrays.

Next: Project: A Markdown-to-HTML Converter with sed and AWK