Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 37 ยท IDS/IPS and Monitoring

Log Analysis and SIEM

Learn how network and firewall logs become security alerts: collecting, parsing, normalizing, correlating events in a SIEM, and detecting brute force and port scans in code.

A sensor that raises an alert nobody sees protects nothing. Every firewall, router, server and IDS produces logs, and during an incident those logs are usually the only record of what happened. Log analysis is the skill of turning millions of lines into a few sentences: who did what, when, from where, and whether it matters. A SIEM (Security Information and Event Management system) is the platform that collects, normalizes, stores, searches and correlates those logs at scale.

This lesson walks through the pipeline and then does the real work in code: parse log lines, detect password guessing, and detect a port scan by correlation. All data is invented.

The SIEM pipeline

flowchart LR
    S["Sources<br/>firewall, IDS, servers, DNS"] --> C["Collect<br/>agents, syslog, APIs"]
    C --> P["Parse + normalize<br/>common fields"]
    P --> St["Store + index"]
    St --> R["Correlate + detect"]
    R --> A["Alert / ticket"]
    A --> I["Investigate + respond"]

Each stage can fail silently, so monitor the pipeline itself.

  1. Collect. Syslog over TCP/TLS, agents on hosts, cloud APIs. Use reliable transport and protect the channel: logs are evidence.
  2. Parse and normalize. Every product spells things differently (src, SourceAddress, client_ip). Normalization maps them to one schema (source.ip) so one query works across vendors.
  3. Store. Hot storage for recent, fast search; cheaper archive for long retention required by regulation.
  4. Correlate. Combine events: ten failed logins followed by a success from the same address is far more meaningful than either alone.
  5. Alert and respond. Alerts go to analysts or automated playbooks (for example, block the address). See incident response.

Time is everything

Correlation needs events in the right order, so clocks must be synchronized (NTP, ideally from a trusted internal source) and logs should carry time zones, preferably UTC. If a firewall is 7 minutes ahead of a server, the attack timeline becomes nonsense.

Parsing a log line

Here is a firewall-style line, invented for this lesson:

TEXT
2026-03-14T10:22:41Z fw01 DENY tcp 203.0.113.50:51022 -> 192.0.2.10:22 flags=S

Fields: time, device, action, protocol, source, destination, TCP flags. Parsing means extracting these into named fields. Regular expressions with named groups are the standard tool.

Python Parse firewall log lines into fields

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Always count the unparsed lines. A sudden jump in unparsed lines usually means a device changed its format and your detections have gone blind.

Detection 1: brute-force logins

Password guessing leaves a recognizable trail: many failures from one source in a short time, sometimes followed by a success. The rule below is a typical SIEM correlation, written plainly.

Python Failed logins, then a success

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first address is flagged critical: failures against several accounts and then a success suggests a guessed credential. The second address with one typo is ignored. Notice how severity comes from correlating two event types.

Detection 2: port scans

A host probing many different ports or addresses, with mostly blocked results, is performing reconnaissance. Count the distinct destination ports per source.

Python Distinct ports per source

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real detectors also look for horizontal scans (one port across many hosts) and slow scans spread over hours, which need longer windows.

Searching and hunting

Beyond fixed rules, analysts hunt: they form a hypothesis ("an attacker is using DNS tunnelling") and search for evidence. Useful habits:

  • Start from a known-bad indicator (an IP, a domain, a file hash) and pivot: who else talked to it?
  • Look at rare things: the least common user agent, the rarest destination country, the process that talks to the internet once a day.
  • Compare against the baseline.
  • Write down what you searched even when you found nothing; that negative result is useful evidence.

Keeping logs trustworthy

  • Integrity: ship logs off the machine quickly; an attacker with root will try to wipe local logs. Use append-only or write-once storage for important sources.
  • Completeness: log successes as well as failures, and include DNS, DHCP, proxy and VPN logs. They are what let you map an IP address at 10:22 to a named user.
  • Privacy and retention: logs contain personal data; define retention, access and masking rules.
  • Volume control: do not log everything at debug level; decide which events earn their storage cost.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does normalization do in a SIEM pipeline?

    Normalization lets one query or rule work across logs from many different products.
  2. 2. Why must clocks be synchronized across devices?

    Correlation and incident timelines depend on trustworthy timestamps.
  3. 3. Which pattern most strongly suggests a successful password-guessing attack?

    Failures followed by a success from the same source suggests a guessed credential.
  4. 4. A sudden rise in 'unparsed' log lines most likely means:

    Parsers rely on a known format; when it changes, events silently stop matching.
  5. 5. Why ship logs off the host quickly?

    Off-host copies preserve evidence even if the attacker wipes the machine.

Summary

  • A SIEM collects, parses, normalizes, stores, correlates and alerts on logs from many sources.
  • Accurate, synchronized time (UTC) is essential for correlation.
  • Parse with named regular expressions and monitor the count of unparsed lines.
  • Correlate events (failures then success, many ports from one source) to raise precision and severity.
  • Hunt with hypotheses and pivots, and preserve log integrity and privacy.

Next: Wi-Fi Security: WPA2, WPA3 and Rogue Access Points