Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 47 ยท Projects, Practice and Review

Project: Web Log Analysis with MapReduce

Build a complete MapReduce pipeline that parses web server access logs, finds top pages and errors per hour, and uses a combiner to cut shuffle data.

Web server logs are the classic big data input: one line per request, billions of lines per month, and questions such as "which pages are most popular?" or "when do errors spike?" In this project you will answer three such questions with MapReduce jobs: top pages, errors per hour and a status-code summary, and you will add a combiner and measure what it saves.

The real jobs would run as Java or Hadoop Streaming programs on a cluster (see Writing a MapReduce Job in Java and Hadoop Streaming with Python). Here a small Python simulator plays the framework: it splits the input, runs mappers, partitions and groups the pairs, and runs reducers. The logic of each mapper and reducer is exactly what you would deploy; the cluster is the stand-in. Real log files would be read from HDFS, not from a list in memory.

The plan

Three questions, one pattern

Top pages Count hits per URL, keep the biggest.
Errors per hour Filter 5xx, count per hour.
Status mix Count per status class.
1 / 2

All three jobs are variations of word count: the key is what you group by (URL, hour, status class) and the value is 1. If that sounds familiar, you have seen it in The MapReduce Programming Model; the new parts are parsing real-looking text and handling bad lines.

Step 1: Parse the log format

Most web servers write the "combined" format. One line looks like this:

TEXT
203.0.113.9 - - [12/Mar/2026:14:07:31 +0000] "GET /products/42 HTTP/1.1" 200 5120

The fields are client IP, two ignored fields, the timestamp, the request (method, path and protocol), the status code and the response size in bytes. A regular expression extracts them. Real logs always contain garbage lines (truncated writes, bots sending nonsense), so the mapper must skip lines it cannot parse and count them, never crash. In Hadoop you count with a counter; the simulator has one too.

Python Parse access-log lines safely

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: A tiny MapReduce framework

The simulator needs five ideas: input splits, a mapper, an optional combiner, a partitioner and a reducer. Two details matter. First, the combiner runs per split (per mapper), so it only merges pairs from the same mapper. Second, we count the pairs that cross the shuffle to see the savings, as in Inside the Shuffle and Sort.

Step 3: Top pages and a combiner

The mapper emits (path, 1) for every successful or failed request. The reducer sums. To find the top N we take the reducer output, which is small compared with the input, and sort it in a last step (a single reducer, or the driver program, is fine for that).

Python Top pages, with and without a combiner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check that both runs return the same counts: the combiner changes only how much data travels. It is valid here because addition is associative and commutative.

Step 4: Errors per hour

Now filter and group by a different key. The mapper emits ("14", 1) only when the status is 500 or higher. Filtering in the mapper is the cheapest place to throw data away, because filtered records never touch disk or network. The key is a string hour so keys sort in time order ("03" before "14"; zero-padding matters).

If you also want the error rate, you need two numbers per hour, errors and total. Emit a pair (errors, total) as the value, and let the combiner and reducer add both components. Never emit a ratio from the mapper (ratios cannot be summed; the average problem from the shuffle lesson).

Python Errors per hour and error rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 5: What would change on a real cluster?

  • Input splits follow HDFS blocks, so a 100 GB log directory gives about 800 mappers with 128 MB blocks. Log files should be rolled and compressed with a splittable choice (or stored as many moderate-size files) so they parallelize.
  • Bad lines are counted with context.getCounter(...); the job report shows how many were skipped. Fail the job only if the share of bad lines passes a threshold.
  • Time zones: logs carry an offset (+0000). Convert to a single zone (usually UTC) in the mapper, before building the hour key.
  • Top N over a huge key space uses a per-mapper top-N, then a single reducer merges them (see MapReduce Design Patterns).
  • Output format: write the results as Parquet or a Hive table partitioned by day, so analysts can query them (see Partitioning, Bucketing and File Layout).

Extensions to try

  1. Count unique IP addresses per page (emit (path, ip), deduplicate in the reducer).
  2. Add bytes served per hour, using a sum in both combiner and reducer.
  3. Identify bots by user agent. This needs the extra fields from the combined format; extend the regular expression.
  4. Turn the mapper and reducer into two stdin/stdout scripts for Hadoop Streaming.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What should a log-parsing mapper do with a line that does not match the expected format?

    Real logs contain garbage. Skipping with a counter keeps the job running and still shows how many lines were lost.
  2. 2. Why is filtering 5xx errors inside the mapper better than filtering in the reducer?

    Dropping records early saves disk writes and network transfer. The shuffle is the expensive part.
  3. 3. How do you compute an error rate per hour correctly with a combiner?

    Ratios cannot be combined directly. Sums of counts are associative, so divide only after the final reduction.
  4. 4. Why does the combiner not change the final answer for page counts?

    Summing partial sums gives the same total, so combining on the map side only reduces how much data is sent.

Summary

  • Log analysis jobs are word count with a parsed key: URL, hour or status class.
  • Parse defensively: skip and count bad lines, normalize time zones in the mapper.
  • Filter early in the mapper; the shuffle is where cost is.
  • A combiner is safe for sums and counts; carry (numerator, denominator) pairs for rates.
  • Top N is a small final step because the aggregated result is much smaller than the input.

Next: Project: Search Index over Documents.