Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Projects and Practice

Project: Web Server Log Analysis

Parse raw web server log lines with Polars string expressions and regex, then find top pages, error rates per minute, slow endpoints, busy clients and user sessions.

Log files are one of the most common "big text" data sources: every request to a web server becomes a line of text. Turning those lines into a table lets you answer operational questions quickly: which pages are slowest, when did errors spike, who is hammering the server, how long do visits last?

In this project the raw material is a column of text lines. You will parse it with Polars string expressions and regular expressions (no loops), convert the pieces to proper types, and then analyze it with group_by, group_by_dynamic, quantiles, and window functions. Everything is deterministic: we generate the log ourselves with a seeded random generator.

Step 1: generate raw log lines

Real files look like the following. We create 3,000 lines in Python, as one list of strings, and put them in a one-column DataFrame, exactly as pl.read_csv(..., separator="\n") or a text read would.

TEXT
203.0.113.7 - [2025-03-14 09:15:02] "GET /products/42 HTTP/1.1" 200 5123 0.084

The fields are: client address, timestamp, request line, status, bytes, and response time in seconds.

Python Create the raw log

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: parse with a regular expression

str.extract_groups applies one regular expression with several named groups and returns a struct column; unnest then turns the struct into normal columns. This is the cleanest way to split a line into fields.

Python Extract fields with named groups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If a line does not match, every group is null (not an error), so counting nulls tells you how many malformed lines the file has. That last line is the first thing to check in any parsing job: "did everything parse?" Dirty logs always contain a few surprises.

Regular expressions in Polars follow the Rust regex syntax, which is fast and safe (no catastrophic backtracking) but does not support look-around or backreferences. Use str.extract with a group number for one field, str.extract_groups for many.

Step 3: first questions

With a typed table, the common questions are short queries.

Python Top pages, status mix, slowest endpoints

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Averages hide slow requests; percentiles (median, 95th) show what users actually feel. Here /checkout stands out on both latency and error rate, because we built it to be the weak spot. The boolean-mean pattern (pl.col("status") >= 500).mean() is the error rate per endpoint in one expression.

Step 4: errors over time

When did things go wrong? group_by_dynamic buckets the sorted timestamp column into fixed windows.

Python Requests and errors per five minutes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The data must be sorted by the time column first (log.sort("ts")); the window labels are the start of each five-minute bucket. In production you would raise an alert when error_pct crosses a threshold, and you can change every="5m" to "1m", "1h" or "1d" to zoom in or out.

Step 5: busy clients and sessions

A session is a run of requests from one client where no gap is longer than, say, ten minutes. That is a classic window-function problem: sort by client and time, compute the gap to the previous request, mark a new session wherever the gap is large, and take a cumulative sum of those marks.

Python Sessionize requests per client

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three steps are the standard recipe, worth memorizing: diff, flag, cumulative sum. The first request of each client has no previous time (gap is null), which also counts as a session start.

Extensions to try

  • Detect bots: clients with more than N requests per minute (use group_by_dynamic with a group_by="ip" argument).
  • Compute the bytes per status class and the share of traffic spent on errors.
  • Find entry pages (the first path of each session) and exit pages (the last).
  • Normalize paths such as /products/42 into /products/{id} with str.replace(r"/\d+", "/{id}") before grouping.
  • Write the parsed table to Arrow IPC and reload it to see how much faster it is than re-parsing text (IO chapter).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does str.extract_groups(pattern) return for a line that does not match the regex?

    Unmatched lines yield null groups, so counting nulls tells you how many lines failed to parse.
  2. 2. Why report the 95th percentile of response time rather than only the mean?

    Latency distributions are skewed, so averages understate slow experiences.
  3. 3. What must be true of the time column before group_by_dynamic?

    Dynamic windows are built in one ordered pass, so the column must be sorted.
  4. 4. Which three steps build sessions from timestamps per client?

    Each flagged gap starts a new session number; cum_sum over the flags assigns a session id inside each client.

Summary

  • Put raw lines in a one-column frame and parse with str.extract_groups, then unnest.
  • Cast parsed text to real types; count null groups to find malformed lines.
  • Use boolean means for rates and quantile for latency percentiles.
  • group_by_dynamic buckets sorted timestamps into windows.
  • Sessions come from the diff, flag and cumulative-sum recipe inside over.

Next: Project: Sensor Time Series