Skip to content
Tutorials Center
My learning

Topics

Lesson 42 of 48 ยท Projects

Project: Web Server Log Analysis

Build a log analysis job step by step: parse access logs, count bad lines, rank top URLs, find error spikes and write results, with PySpark code and runnable stand-ins.

Web server logs are the classic first Spark project. They are large, text-based, messy and full of questions: which pages are most popular, when do errors spike, which clients hammer the API? One busy site can produce gigabytes of access logs per hour, which is why this job is worth running on a cluster instead of with grep and sort.

In this project you build the analysis in four steps: parse each line into fields, count the lines that fail to parse, aggregate by status, URL, hour and client, and write the results for a dashboard. The real PySpark code is shown read-only because Spark cannot run on this site. Next to it, every step has a runnable stand-in in plain Python, using the same logic and the same shapes (lines in, key-value pairs, counts out), so you can run it, change it and see the answers. This project combines RDD transformations, pair RDDs, DataFrame aggregations and accumulators.

The data and the plan

We use the Common Log Format that Apache and nginx can write. One request is one line:

TEXT
203.0.113.7 - - [10/Oct/2026:13:55:36 +0000] "GET /products/42 HTTP/1.1" 200 2326

The fields are the client IP, two identity fields that are almost always -, the timestamp in brackets, the request in quotes (method, path, protocol), the HTTP status code and the number of bytes sent (or - when none).

flowchart LR
    A["Raw log files"] --> B["Parse with regex"]
    B -->|"no match"| C["Bad-line counter"]
    B -->|"fields"| D["Clean types:<br/>status, bytes, hour"]
    D --> E["Aggregate:<br/>status, URLs, hours, IPs"]
    E --> F["Write results"]

Lines that do not match the pattern are counted and set aside, never allowed to crash the job.

The same plan in PySpark has three parts. Read the files as plain text (one row per line, in a column called value), extract the fields with regexp_extract, and aggregate:

Python
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("log-analysis").getOrCreate()

PATTERN = r'^(\S+) \S+ \S+ \[([^\]]+)\] "(\S+) (\S+) \S+" (\d{3}) (\d+|-)$'

raw = spark.read.text("/data/access/*.log")

parsed = raw.select(
    F.regexp_extract("value", PATTERN, 1).alias("ip"),
    F.regexp_extract("value", PATTERN, 2).alias("ts"),
    F.regexp_extract("value", PATTERN, 3).alias("method"),
    F.regexp_extract("value", PATTERN, 4).alias("path"),
    F.regexp_extract("value", PATTERN, 5).alias("status"),
    F.regexp_extract("value", PATTERN, 6).alias("bytes"),
)

regexp_extract returns an empty string when the pattern does not match, so a row with status == "" is a bad line. This is a useful property: nothing throws, and we can count the failures with a plain filter.

Step 1: Parse the lines

The first stand-in does the same parse in Python on a few hand-written lines, two of which are broken on purpose. It is the same regular expression as above.

Python Parse log lines and keep the bad ones

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the bytes field: - means "nothing sent", so it becomes 0 and not a failed conversion. Cleaning such small details at parse time keeps the rest of the pipeline simple. In PySpark you would write F.when(F.col("bytes") == "-", 0).otherwise(F.col("bytes").cast("long")).

Step 2: Count what you could not parse

A job that silently drops 30% of its input is worse than one that fails. Always report how many lines were rejected. With DataFrames it is one filter and a count; with RDDs an accumulator counts rejected lines while the main work runs:

Python
total = parsed.count()
bad = parsed.filter(F.col("status") == "").count()
print(f"{bad} of {total} lines did not match ({bad / total:.2%})")

good = (parsed
        .filter(F.col("status") != "")
        .withColumn("status", F.col("status").cast("int"))
        .withColumn("bytes", F.when(F.col("bytes") == "-", 0).otherwise(F.col("bytes").cast("long")))
        .withColumn("time", F.to_timestamp("ts", "dd/MMM/yyyy:HH:mm:ss Z"))
        .withColumn("hour", F.date_trunc("hour", "time")))
good.cache()          # we run several aggregations on it

cache() matters here because each aggregation below is a separate action that would otherwise re-read and re-parse every file (see Lazy Evaluation, Transformations and Actions). A threshold is a good production habit: fail the job if more than, say, 1% of lines are bad, because that usually means the log format changed.

Step 3: Aggregate

The questions we want to answer, each with a one-line DataFrame and a stand-in:

Python
# Requests per status class
good.groupBy((F.col("status") / 100).cast("int").alias("class")).count().orderBy("class").show()

# Top 10 URLs
good.groupBy("path").count().orderBy(F.desc("count")).limit(10).show()

# Error rate per URL, only for URLs with enough traffic
(good.groupBy("path")
     .agg(F.count("*").alias("requests"),
          F.sum(F.when(F.col("status") >= 500, 1).otherwise(0)).alias("errors"))
     .filter("requests >= 100")
     .withColumn("error_rate", F.col("errors") / F.col("requests"))
     .orderBy(F.desc("error_rate"))
     .show())

# Busiest clients and traffic per hour
good.groupBy("ip").agg(F.count("*").alias("hits"), F.sum("bytes").alias("bytes")).orderBy(F.desc("hits")).show(10)
good.groupBy("hour").agg(F.count("*").alias("requests")).orderBy("hour").show()

The stand-in below generates 5,000 synthetic log lines with a fixed random seed, so you get the same numbers on every run. It models the RDD style: lines flow through map and filter, become (key, 1) pairs and are summed per key, the way reduceByKey does. The checkout page is built to fail more often than the rest, which is the kind of discovery log analysis exists for.

Python Top URLs, status classes and error rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error table makes the problem obvious: /checkout fails about 31 percent of the time while the other pages stay between 0.4 and 1.4 percent. In a real investigation you would now drill into that URL by hour, by client and by status code.

Step 4: Time and clients

Two more views finish the dashboard: requests and errors per hour to find spikes, and the heaviest clients to spot crawlers or abuse. A client with a hit count far above the others may need rate limiting.

Python Errors per hour and the busiest clients

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One client generating more than half of all traffic is a classic bot signature, and also a classic skew problem: a groupBy("ip") would send most rows to a single task (see Testing, Debugging and the Spark UI for how that looks in the UI).

Write the results

Dashboards should read small result tables, not raw logs. Write each aggregate as its own output, partitioned by date so that tomorrow's run adds a folder instead of rewriting everything:

Python
daily = (good.withColumn("date", F.to_date("time"))
             .groupBy("date", "path")
             .agg(F.count("*").alias("requests"),
                  F.sum(F.when(F.col("status") >= 500, 1).otherwise(0)).alias("errors")))

(daily.write
      .mode("overwrite")
      .partitionBy("date")
      .parquet("/data/reports/url_daily"))

With spark.sql.sources.partitionOverwriteMode set to dynamic, mode("overwrite") replaces only the date folders present in the data, so a re-run of one day is safe. Notice that this project is idempotent: the same input always produces the same output, so a failed run can simply be repeated.

Real versus stand-in. The PySpark code above is the real solution but did not run here. The stand-ins use Python's re, Counter and loops on 5,000 synthetic lines. They match the logic of regexp_extract, filter, groupBy().count() and orderBy, but on a real cluster the same code reads many files in parallel, shuffles by key between stages and can process billions of lines.

Ideas to extend it

  • Parse the user agent and referrer (the combined log format has two more quoted fields) and find the share of mobile and bot traffic.
  • Use a window function to find, per client, the gap between consecutive requests, and group requests into sessions with a 30-minute inactivity rule.
  • Join with a table of known bot IP ranges, using a broadcast join (see Joins).
  • Switch the source from files to a stream and compute the same counts with windows and watermarks.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does regexp_extract return when the pattern does not match a line?

    regexp_extract gives an empty string on no match, so rows with an empty status can be counted and filtered as bad lines.
  2. 2. Why call cache() on the parsed DataFrame in this project?

    Every action recomputes its lineage. Caching the parsed data lets all the aggregations reuse it.
  3. 3. In the CLF line the status is 404 and the bytes field is a dash. How should bytes be handled?

    A dash means no bytes were sent. Casting it directly would produce null or an error, so convert it to 0 explicitly.
  4. 4. One client IP accounts for more than half of all requests. What is a risk when grouping by IP in Spark?

    A very frequent key sends most rows to one partition and one task, which then becomes the straggler of the stage.

Summary

  • A log analysis job parses lines with a regex, separates and counts bad lines, then aggregates by status, URL, hour and client.
  • regexp_extract returns an empty string on no match, so failed parses are easy to count and filter without exceptions.
  • Cache the cleaned data when several aggregations share it, and write small, date-partitioned result tables for dashboards.
  • Look for skew: a few URLs or clients can dominate, and one very frequent key makes one task do most of the work.
  • Keep jobs idempotent so a failed day can be re-run safely.

Next: build a layered data pipeline in Project: A Bronze-Silver-Gold ETL Pipeline.