Skip to content
Tutorials Center
My learning

Topics

Lesson 55 of 80 ยท Security Operations and Incident Response

Logging, Log Analysis and SIEM

Learn what to log, common log formats, how to parse logs with regex and pandas, aggregate events and write alert rules, plus how a SIEM ties it all together.

When something goes wrong, the first questions are always the same: what happened, when, to whom, and from where? Without logs you can only guess. With good logs you can answer, often within minutes. Logs are the memory of your systems. They are also the raw material for every detection you saw in Detecting Malicious Behavior: a rule can only fire on an event somebody recorded.

In this lesson you'll learn what security teams log and why, how common log formats look, how to turn raw lines into structured data with regular expressions and pandas, how to aggregate and write simple alert rules, and what a SIEM adds on top. All logs here are invented. Real ones need the same steps, only at far larger volume.

What to log

You can't keep everything, and logging too much is its own problem (cost, noise, privacy). Focus on events that answer security questions:

Area Examples Why it matters
Authentication logins, failures, logouts, MFA prompts, password resets Brute force, stolen credentials
Authorization access denied, permission and role changes Privilege abuse, probing
Administrative actions new accounts, config changes, policy edits Insider threats, persistence
Application events input validation failures, errors, payments, exports Attacks against the app
Network and DNS firewall allow/deny, DNS queries, proxy requests, flows Command and control, data theft
Endpoint process starts, scheduled tasks, service installs, USB use Malware behavior
Cloud and infrastructure API calls, key use, storage access Misconfiguration and abuse

Each log entry should carry the essentials: when (timestamp with time zone, preferably UTC), who (user or service), where (source IP, host), what (the action and the object) and outcome (success or failure). A request ID that appears in every system a request touches makes it possible to follow one action across services.

What not to log

Never write passwords, session tokens, API keys, full card numbers or other secrets into logs, and think about personal data: logs often live longer and are read by more people than the database they describe. Neutralize user-controlled text before it is written, because an attacker can inject a newline to forge a fake log line (log injection). See Secrets, Logging and Error Handling and Privacy and Data Protection.

Trustworthy logs

Logs are evidence, so protect them. Send them off the machine quickly to a central store where an intruder who owns the server cannot erase them, synchronize clocks (NTP) so events line up across systems, restrict who can read and delete them, and set a retention period that covers your investigation and legal needs.

Log formats you will meet

  • Syslog lines: Oct 12 03:14:07 web1 sshd[812]: Failed password for root from 203.0.113.9 port 51234 ssh2.
  • Web server access logs, usually in the combined format: 203.0.113.9 - - [12/Oct/2025:03:14:07 +0000] "GET /login HTTP/1.1" 401 512.
  • Key=value lines: ts=2025-10-12T03:14:07Z user=ana action=login result=fail ip=203.0.113.9.
  • JSON: one JSON object per line. This is the best choice for new systems because every field is named and nothing needs guessing.
  • Windows Event Log and cloud audit logs, which are structured records with event IDs.

The trouble with free text is that every program formats it differently. The job of log analysis often starts with parsing: turning a text line into named fields. Structured logging (JSON or key=value from the start) removes most of that pain.

Parsing with regular expressions

A regex with named groups is a parser for one specific format. The demo below handles a few combined-format web log lines (invented, using documentation IP ranges). Lines that do not match are counted instead of crashing the program, which matters because real logs always contain surprises.

Python Parse web access logs with a regex

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Regular expressions are fragile: a change in the log format silently breaks the parser. Always count the lines you could not parse and alert when that number rises.

Aggregating with pandas

Once you have records, questions become table operations. pandas is the usual tool for analysis of a log extract that fits in memory. The next demo builds a seeded synthetic hour of traffic, loads it into a DataFrame and answers three common questions: which status codes dominate, which IPs make the most failed logins, and which paths return the most errors. The data is generated, not real.

Python Aggregate synthetic logs with pandas

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One IP stands out in the second table, and the third shows when the burst happened. Without aggregation you would be scrolling through hundreds of lines. This pattern, "filter, group, count, sort", is the core of nearly all log analysis. It is also what SIEM query languages do, in a different syntax.

From counts to alert rules

An alert rule turns a pattern into a notification. A good one is specific, explains why it fired and tells the reader what to do next. Common building blocks:

  • Threshold: more than N events in a time window ("10 failed logins from one IP in 5 minutes").
  • Sequence: event A followed by event B ("several failures, then a success from the same IP").
  • Rarity: something that has never happened before ("first login from a new country").
  • Match: an IOC or a forbidden action ("access to a honeypot file").

The demo below applies a threshold rule and a sequence rule to a small list of authentication events in a sliding time window. Because it tracks failures before a success, it catches the dangerous moment when a guess finally works.

Python Threshold and sequence alert rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The honest user ana and dana caused no alert, even though dana failed once. Notice the choices you must make: window length, threshold and severity. They are guesses at first, improved by looking at the alerts that turn out to be false. A rule that fires a hundred times a day on harmless activity trains analysts to ignore it, a problem known as alert fatigue.

What a SIEM adds

A SIEM (security information and event management) system is the central place where logs go to be searched, correlated and alerted on. Products differ, but the functions are consistent:

flowchart LR
    A["Servers, endpoints,<br/>firewalls, cloud, apps"] --> B["Collect<br/>(agents, syslog, APIs)"]
    B --> C["Normalize<br/>(common field names)"]
    C --> D["Store and index<br/>(search, retention)"]
    D --> E["Correlate and detect<br/>(rules, analytics)"]
    E --> F["Alert, dashboard,<br/>case management"]
    F --> G["Analyst investigates<br/>and responds"]

A SIEM is a pipeline: collect, normalize, store, detect, respond.

  • Normalization maps different formats to common names (src_ip, user, action), so one rule works on logs from a firewall and a VPN alike.
  • Correlation joins events from different sources: a VPN login from an unusual country, followed by a new admin account in the directory, followed by a large download from a file server.
  • Enrichment adds context: threat-intelligence matches, asset owners, geolocation, whether the user is on leave.
  • Retention and search let you investigate old incidents and satisfy compliance rules (see Compliance and Security Frameworks).
  • SOAR (security orchestration, automation and response) tools sit beside or inside a SIEM to run playbooks, for example disabling an account or isolating a host when a rule fires.

Related terms you will hear: XDR (extended detection and response, a vendor-integrated take on the same idea), UEBA (analytics based on user and entity baselines) and the SOC (security operations center), the team that watches the alerts. Free and commercial tools exist at every scale, from a small open-source stack to a cloud service. For a small team, a well-run central log store with a few good alerts beats an expensive platform nobody has time to tune.

Practical advice

  1. Start with questions, not volume. Decide which attacks matter and log the evidence you would need.
  2. Standardize fields and use UTC everywhere.
  3. Prefer structured logs and a shared schema.
  4. Alert on outcomes you would act on. If nobody would respond, it is a report, not an alert.
  5. Review rules regularly. Remove noisy ones, add ones for new threats, and test them.
  6. Check the pipeline. Alert if a source stops sending logs, because silence can mean an attacker switched off logging.

The next lesson applies these ideas to concrete attack patterns: Spotting Attacks in Logs.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which of these must NEVER be written to application logs?

    Logs are widely readable and kept for a long time. Secrets and tokens in logs give an attacker who reads them direct access.
  2. 2. Why is it recommended to ship logs to a central, separate system quickly?

    Local logs can be deleted or altered by an intruder with enough access. A separate, restricted store preserves the evidence.
  3. 3. What does a SIEM's normalization step do?

    Firewalls, servers and applications name fields differently. Normalizing them lets one correlation rule work on all of them.
  4. 4. A rule fires hundreds of times a day, almost always on harmless events. What is the main risk?

    Noisy alerts erode attention. Tune thresholds, add context or retire the rule so that each alert deserves a response.
  5. 5. Which alert is a sequence rule?

    A sequence rule links events in order: failures first, then a success. Threshold rules count, and match rules compare against a list.

Summary

  • Logs answer who did what, when, from where and with what result; log authentication, authorization, admin actions, application, network, endpoint and cloud events.
  • Never log secrets, protect logs from tampering, use UTC and synchronized clocks, and ship logs to a separate store.
  • Structured formats such as JSON are easy to analyze; for free text, parse with named-group regexes and count lines that fail to parse.
  • pandas turns logs into answers with filter, group, count and sort; alert rules use thresholds, sequences, rarity and matches.
  • A SIEM collects, normalizes, stores, correlates and alerts, but only pays off when rules are tuned and alerts have an owner.

Next: Spotting Attacks in Logs, where you'll look for brute force, impossible travel and odd-hour activity.