Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 38 ยท Projects and Practice

Project: A Log File Analyzer in Groovy

Build a log analyzer in Groovy: parse access-log lines with regex and named groups, group and count with collections, report top pages and error rates, and write JSON.

Log files are where Groovy's text handling, regular expressions and collection methods pay off. In this project you read an access log, turn each line into a small object, and answer questions: which pages are most popular, how many requests failed, which hour is busiest, what are the slowest URLs? Then you export the report as JSON. The same approach works for application logs, CSV exports and any line-oriented text.

Groovy cannot run on this site, so the Groovy code is read-only. The Java program near the end runs the same pipeline on sample lines, so you can check the numbers.

The data

A line in the common access-log style:

TEXT
203.0.113.7 - - [10/Mar/2025:13:55:36 +0000] "GET /index.html HTTP/1.1" 200 2326 0.012
198.51.100.4 - - [10/Mar/2025:13:56:01 +0000] "POST /api/login HTTP/1.1" 401 87 0.230
203.0.113.7 - - [10/Mar/2025:14:02:10 +0000] "GET /missing HTTP/1.1" 404 512 0.004

The fields are client address, timestamp, request line, status, bytes sent and (in our variant) the time in seconds.

Step 1: parse a line with a regex

Groovy has a slashy-string syntax and the =~ (find) and ==~ (match) operators.

GROOVY
import groovy.transform.Canonical

@Canonical
class Entry {
  String ip
  String time
  String method
  String path
  int status
  long bytes
  double seconds

  int hour() { time.split(':')[1] as int }
  boolean isError() { status >= 400 }
}

def LINE = /^(\S+) \S+ \S+ \[([^\]]+)\] "(\w+) (\S+) [^"]*" (\d{3}) (\d+) ([\d.]+)$/

Entry parse(String line) {
  def m = line =~ LINE
  if (!m.matches()) return null
  def (all, ip, time, method, path, status, bytes, secs) = m[0]
  new Entry(ip, time, method, path, status as int, bytes as long, secs as double)
}

Details:

  • A slashy string /.../ needs no doubled backslashes, which suits regexes.
  • line =~ LINE creates a Matcher. m.matches() checks the whole line, and m[0] is a list: the whole match followed by each capture group, which a multiple assignment def (all, ip, ...) = m[0] unpacks.
  • Conversions use as int, as long, as double.
  • Lines that do not match return null and are counted as malformed instead of crashing the run.

Step 2: read the file lazily

GROOVY
def entries = []
int bad = 0
new File("access.log").eachLine { line ->
  def e = parse(line)
  if (e) entries << e else bad++
}
println "Parsed ${entries.size()} lines, skipped ${bad}"

eachLine reads one line at a time and works for files of any size. For huge logs, do the counting inside eachLine instead of keeping every entry.

Step 3: ask questions with collection methods

GROOVY
// top pages
def topPages = entries.countBy { it.path }
  .sort { -it.value }
  .take(5)

// status classes
def byClass = entries.countBy { "${it.status.intdiv(100)}xx" }

// error rate in percent
def errorRate = entries.count { it.isError() } * 100.0 / entries.size()

// requests per hour
def perHour = entries.countBy { it.hour() }.sort()

// slowest urls by average time
def slowest = entries.groupBy { it.path }
  .collectEntries { path, list -> [path, list*.seconds.sum() / list.size()] }
  .sort { -it.value }
  .take(3)

// total traffic per client in KB
def traffic = entries.groupBy { it.ip }
  .collectEntries { ip, list -> [ip, list*.bytes.sum() / 1024] }

Remember what each method returns: countBy gives a map from key to count, groupBy a map from key to list, collectEntries builds a map from pairs, sort { } on a map sorts by its entries, and take(n) keeps the first n. These are the same methods you learned in the collections chapter, now used on real data.

Step 4: print and export

GROOVY
println "Top pages:"
topPages.each { path, n -> printf("  %-20s %5d%n", path, n) }
println "Error rate: ${String.format('%.1f', errorRate)}%"

import groovy.json.JsonOutput
new File("report.json").text = JsonOutput.prettyPrint(JsonOutput.toJson([
  total   : entries.size(),
  skipped : bad,
  errorRate: errorRate,
  topPages: topPages,
  perHour : perHour
]))

printf uses format specifiers (%-20s left-aligned string, %5d right-aligned integer). JsonOutput.toJson accepts maps and lists, and prettyPrint formats the string.

Making it robust

  1. Time zones and dates: parse the timestamp with java.time.OffsetDateTime and a DateTimeFormatter.ofPattern("dd/MMM/yyyy:HH:mm:ss Z", Locale.ENGLISH) instead of splitting text.
  2. Multiple files and .gz: new GZIPInputStream(new FileInputStream(f)).eachLine { }.
  3. Streaming statistics: keep counters in maps and update them per line, so memory use stays flat.
  4. Tests: write a Spock where: table of sample lines and expected fields (see data-driven testing).
  5. Security: logs contain user-controlled text. Escape it before putting it in HTML.

Try it: the same pipeline in Java (stand-in)

The Java program parses the three sample lines with a regex (named groups) and prints the report.

Java Stand-in: parse and summarize log lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Try it: counting with a plain map (stand-in)

countBy is a loop with a map. Writing it by hand shows what the Groovy method hides.

Java Stand-in: what countBy does

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does def (all, ip, time) = m[0] do when m is a Matcher with groups?

    m[0] is a list of the full match followed by each group, so multiple assignment unpacks it.
  2. 2. What does countBy { it.path } return?

    countBy groups items by the closure's result and counts them.
  3. 3. Why read the file with eachLine?

    eachLine streams the file, which scales to logs of any size.
  4. 4. How should malformed lines be handled?

    Counting skipped lines keeps the run going and tells you how clean the data was.

Summary

  • A regex with capture groups turns each line into an object; slashy strings avoid backslash doubling.
  • eachLine streams files; countBy, groupBy, collectEntries, sort and take answer the questions.
  • Report with printf and export with JsonOutput.
  • Make it robust with proper date parsing, streaming counters, tests and escaping.

Next: Project: A Build-Pipeline DSL