Skip to content
Tutorials Center
My learning

Topics

Lesson 71 of 80 ยท Projects

Project: Security Log Analyzer

Build a web-server log analyzer in Python: parse access logs, detect brute force, scanning and injection attempts, then chart the results with pandas.

Every public web server is probed within minutes of going online. Most of the traffic is harmless, some of it is automated noise, and a little of it is aimed at you. The access log records all of it, one line per request, and learning to read that log is one of the most practical defensive skills there is.

In this project you will build a security log analyzer. It parses a web-server access log, turns it into a table, runs three detections (brute-force logins, directory scanning and injection attempts) and produces a chart you could paste into an incident report. The log is synthetic and generated with a fixed seed, so every run gives identical results. The project reuses ideas from Logging, Log Analysis and SIEM, Spotting Attacks in Logs, SQL Injection and Parameterized Queries and Cross-Site Scripting and Output Encoding.

The pipeline

flowchart LR
    G["Raw access log lines"] --> P["Parse with a regex"]
    P --> T["DataFrame<br/>one row per request"]
    T --> D1["Brute force<br/>POST /login failures"]
    T --> D2["Scanning<br/>many 404s"]
    T --> D3["Injection<br/>payload signatures"]
    D1 --> A["Alerts table"]
    D2 --> A
    D3 --> A
    A --> C["Chart and report"]

Parse once into a table, then every detection is a short query over the same data.

The log format

Most web servers write a variant of the combined log format. A line looks like this:

TEXT
203.0.113.77 - - [02/Mar/2026:12:40:02 +0000] "POST /login HTTP/1.1" 401 512 "-" "python-requests/2.31"

From left to right: client address, two fields that are almost always -, the timestamp, the request line (method, target and protocol), the status code, the response size in bytes, the referrer and the user agent. The request target may include a query string (/search?q=blue+shoes), and browsers and attack tools URL-encode special characters there, so a space becomes %20 or + and a quote becomes %27. A detection that searches the raw text for a quote character will miss the encoded form, so the first step after parsing is to decode the target before matching.

Because data in a log is attacker-controlled (they choose the path and the user agent), treat every field as untrusted: never evaluate it, never paste it unescaped into an HTML dashboard and never let a newline in a field forge a new line when you write a log.

Step 1: signatures for injection attempts

Injection attempts have recognizable fingerprints: SQL keywords combined in unusual ways, script tags, and sequences of ../ that try to climb out of a directory. A signature is a regular expression for one such fingerprint. Signatures are cheap and explainable, and they are also noisy, which you will see in a moment. Try them on a few sample targets first.

Python Signature matching on decoded requests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last sample is a false positive: a student searching for a tutorial trips the SQL signature. That is why the final analyzer does not treat one match as an alert. It looks at how many suspicious requests one client sends, whether the server answered with an error, and combines the signal with the other detections.

Attackers also encode payloads twice or use case tricks and comment markers to slip past naive filters, so signature matching is a detection aid, never your defense. The real protection against these attacks is in the application code, as covered in the injection and XSS lessons: parameterized queries and output encoding.

Step 2: the complete analyzer

The program below does everything in one run:

  1. Generates a two-hour synthetic log with ordinary visitors plus three attackers: a password-guesser on /login, a directory scanner and an injection prober. (In real life you would read a file from disk.)
  2. Parses each line with a regular expression into a pandas DataFrame, converts timestamps and decodes targets. Lines that do not match are counted instead of silently dropped, because a parsing gap is itself a blind spot.
  3. Detects:
    • Brute force: ten or more 401 responses to POST /login from one client within five minutes, plus a flag if a successful login follows.
    • Scanning: a client with at least 15 404 responses across at least 10 different paths.
    • Injection: a client with three or more signature hits, or any hit that made the server return a 5xx error. Single hits go in a "review" list.
  4. Reports an alerts table and draws two charts.
Python Security log analyzer: parse, detect, chart

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The report shows the three attackers clearly: the brute-forcer stands out by volume, the scanner by its pile of missing pages, and the injector by its payloads and server errors. The student searching for a tutorial stayed out of the alerts and landed in the review list, which is exactly where a weak, ambiguous signal belongs.

What the analyzer does well, and what it misses

Strengths

  • It is explainable: each alert says why it fired (count, window, distinct paths), so an analyst can verify it in seconds.
  • It uses combinations of signals (a count plus a time window, or a signature plus a 5xx error) to cut false positives.
  • It reports parsing failures instead of hiding them.

Weaknesses you should know about

  • Shared addresses. An office or school behind NAT can look like a single noisy client; mobile carriers do the reverse and spread one attacker over many addresses. Group by account or session as well when you can.
  • Slow attacks. A scanner or guesser that sends one request a minute stays under every threshold. Longer windows and per-account counters help.
  • Encoding tricks. Double encoding, mixed case, comments inside keywords and Unicode variants can defeat signatures. Normalize aggressively before matching, and rely on application-level defenses.
  • POST bodies are invisible. Access logs usually do not record request bodies, so many injection attempts never appear in them. A web application firewall or application logging fills that gap (the next project builds a small one).
  • Logs can lie. A compromised server can alter its own logs. Ship logs to a separate system as they are written.

Ideas for extending the project

  • Add geolocation or ASN data per address and flag logins from unexpected networks.
  • Detect user-agent anomalies (known scanner names, empty agents), keeping in mind they are trivial to fake.
  • Add a sliding-window rate per client and mark bursts of requests per second.
  • Enrich alerts with a risk score and sort the report by it, as you did for vulnerabilities in Vulnerability Management and Patching.
  • Write the alerts as structured JSON lines that a SIEM could ingest.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the analyzer decode URL-encoded targets before matching signatures?

    The server decodes the request before the application sees it, so detections must look at the decoded form too. Raw matching misses quotes and spaces that were percent-encoded.
  2. 2. Which pattern best identifies automated directory scanning?

    A scanner guesses common file names, so it produces a burst of not-found responses for many distinct paths.
  3. 3. Why does the analyzer put a single weak signature match in a review list instead of raising an alert?

    Requiring repeated hits or a server error reduces false positives. Ambiguous cases still reach a human instead of being ignored.
  4. 4. Which limitation is true of detecting injection from web-server access logs alone?

    Typical access logs record the request line, not the body, so payloads in POST data are invisible. Application logs or a WAF are needed to see them.
  5. 5. A log entry's user agent says 'scanner/1.0'. How much should you trust it?

    Every header in a request is chosen by the client. Attackers use browser-like user agents just as easily as scanners advertise themselves.

Summary

  • Parse raw access logs into a table once, then express each detection as a small query.
  • Decode and normalize attacker-controlled fields before matching, count unparseable lines and never trust log content blindly.
  • Brute force is many failed logins in a window (and a success afterwards is the worst case); scanning is many 404s across many paths; injection shows up as payload signatures and server errors.
  • Combine signals, keep a human-review list for weak hits, and accept that signatures are noisy and evadable.
  • Logs complement, not replace, secure code: use parameterized queries, output encoding and a WAF.

Next: Project: A Mini Web Application Firewall, where you will build a rule engine that scores requests and learn why tuning matters.