Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 53 ยท Defense, Detection and Response

Logging, Monitoring and SIEM Basics

Learn what to log, how central collection and a SIEM work, how correlation rules catch attacks, and how to fight alert fatigue with good detection engineering.

You cannot respond to an attack you never noticed. Most breaches that go on for months do so because nobody was looking at the evidence the systems were already producing, or because the evidence was never recorded. Logging is the habit of recording what happened. Monitoring is the habit of reading those records, by people or by software, and acting on them.

This lesson covers what to log, why logs should leave the machine that produced them, how a SIEM (Security Information and Event Management) system correlates events, and how to write detection rules that people trust instead of ignore. A real SIEM cannot run in this tutorial, so product configuration is read-only. The runnable parts are Python stand-ins: small rule engines that work on made-up log lines, showing the logic a SIEM applies at a much larger scale. Earlier lessons on network attacks and authentication flaws describe the behaviours these rules look for.

From event to action

Log Systems record what happened.
Collect Ship logs to one central place.
Correlate Join events into a story.
Alert Tell a human what matters.

A log nobody reads is only a storage bill.

1 / 2

What to log

Start from questions you will need to answer after an incident: who did what, from where, when, and did it work? Events that usually answer them:

  • Authentication. Successful and failed logins, logouts, password changes, MFA failures, account lockouts, and password resets.
  • Authorization and privilege. Role changes, new admin accounts, use of sudo or administrator rights, and access denied responses.
  • Application activity. Requests with status codes, input validation failures, errors, and business events such as "payment refunded" or "export of all customers".
  • System and network. Process starts, service installs, scheduled-task creation, firewall allows and denies, DNS queries, and outbound connections.
  • Configuration changes. Who changed a firewall rule, a cloud permission or a deployment setting.

Every entry should carry a trusted timestamp (UTC, with synchronized clocks), a source (host, service, user, IP address), the action, and the outcome. Log in a structured form such as JSON when you can, so fields can be searched without fragile text parsing.

What not to log

Logs are themselves a target and a liability. Never write passwords, session tokens, full card numbers, private keys or API secrets into a log, and watch for personal data you do not need. A common bug is logging a whole request body that happens to contain a password field. Escape or strip newlines in user-supplied values too, otherwise an attacker can inject a fake line into a text log (log forging).

TEXT
# Dangerous: user input written straight into a text log
2026-03-02T10:15:01Z INFO login failed for user=alice
# An attacker submits the username  "bob\n2026-03-02T10:15:02Z INFO login ok for user=admin"
# and the log now contains a convincing fake success line.

Central collection

If logs stay on the machine that produced them, an attacker who gains control of that machine can delete or edit them. The standard defense is to forward logs off the host as they are written to a central collector that the monitored hosts cannot modify.

flowchart LR
    A["Servers"] --> C["Collector<br/>(agent or syslog)"]
    B["Firewalls and apps"] --> C
    D["Cloud audit logs"] --> C
    C --> N["Normalize and store"]
    N --> R["Correlation rules"]
    R --> T["Alerts and tickets"]

Logs flow from many sources to one place where they are normalized, stored and checked.

Three details make central collection work:

  1. Normalization. Different products describe the same thing differently ("Failed password", event ID 4625, outcome=denied). A SIEM maps them to common fields such as user, src_ip, action and outcome.
  2. Time. Clocks drift. Use NTP everywhere and store UTC, otherwise events from two machines cannot be placed in order.
  3. Retention. Keep enough history to investigate slow attacks. Many intrusions are discovered weeks after they began, so a few days of logs is rarely enough. Decide the retention period from legal duties and incident experience, and protect the archive from modification.

What a SIEM does

A SIEM collects, normalizes, stores and searches logs, and runs detections on them. Its important idea is correlation: combining events that look harmless alone into one meaningful signal.

  • A single failed login is noise. Fifty failed logins for one account from one address within two minutes is a brute-force pattern.
  • Many failures followed by a success from the same address is far more serious, because the guessing may have worked.
  • A login from one country and a second login for the same user from another country ten minutes later is an "impossible travel" signal.
  • A new administrator account created minutes after a login from a never-seen address deserves a human look.

Rules can be threshold-based (more than N events in a window), sequence-based (A then B), allow-list based (anything not on the list), or statistical (far from this user's normal behaviour). Many teams write rules in a vendor-neutral format such as Sigma and convert them to their SIEM's query language, which makes detections easy to share and review.

TEXT
# Read-only: a rule written in plain words
Name:      Possible brute force followed by success
Source:    web authentication logs
Logic:     >= 5 failed logins for one user from one IP in 5 minutes,
           then a success for that user from the same IP
Severity:  high
Response:  lock the account, require a password reset, review the source IP

Detection engineering and alert fatigue

Alert fatigue happens when analysts see so many low-value alerts that they stop trusting any of them. Real attacks then hide in the noise. It is one of the most common reasons monitoring fails in practice. Detection engineering is the discipline of treating rules like software to prevent this: they have owners, tests, version control, review and measured quality.

Habits that keep alerts useful:

  • Start from behaviour, not tools. Map each rule to an attacker technique, for example using the MITRE ATT&CK matrix, and note what you do not cover.
  • Measure precision. For each rule, track how many alerts were real. A rule that is wrong 99 times in 100 needs tuning or removal.
  • Add context. Include the user's role, the asset's importance and previous alerts, so triage takes seconds.
  • Tune by narrowing, carefully. Exclude a known scanner's IP by exact address and write down why, rather than switching a rule off.
  • Test rules. Replay sample logs, including attack samples, and check that the rule fires and does not fire when it should not.
  • Write a runbook. Each alert should link to what the analyst should check and when to escalate to incident response.

Stand-in: a threshold rule

Here is a tiny brute-force detector. It reads made-up authentication events, groups failures per user and source address, and alerts when the count inside a sliding window crosses a threshold. Real SIEMs do the same at scale with indexed data.

Python Brute-force threshold rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What is real and what is the stand-in? Thresholds, windows and keys such as user plus address are real SIEM concepts. The log format, parser and in-memory lists are our simplification. Production systems index events in a database, handle out-of-order arrival, and run rules continuously.

Stand-in: several rules and a noise check

This second example runs three rules against a small set of events and compares the alerts with a list of known-real incidents, measuring each rule's precision (the share of its alerts that were genuine). This is how you decide which rule to tune first.

Python Rules and precision

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the impossible-travel rule fired on the automation account too, which is a classic false positive (service accounts and VPN exits move around). The fix is to scope the rule to human accounts and to document the exception, not to delete the rule.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should logs be forwarded to a central collector soon after they are written?

    Off-host copies preserve evidence even if the attacked machine is compromised. Timestamps and retention still matter.
  2. 2. What is correlation in a SIEM?

    Single events are often harmless. Correlation joins events across time, users and systems to reveal an attack pattern.
  3. 3. Which item should never be written to a log?

    Secrets in logs are exposed to everyone who can read the logs. Record that an event happened, not the secret itself.
  4. 4. A rule is wrong in 99 of 100 alerts. What is the best response?

    Low-precision rules cause alert fatigue. Treat rules like code: measure, tune and test them.

Summary

  • Log authentication, privilege changes, application events, system activity and configuration changes, each with UTC time, source, action and outcome.
  • Never log secrets, and neutralize newlines in user-supplied values.
  • Ship logs off the host to a protected central store with synchronized clocks and sensible retention.
  • A SIEM normalizes events and correlates them using thresholds, sequences, allow-lists or statistics.
  • Fight alert fatigue with detection engineering: behaviour-based rules, measured precision, context, tests and runbooks.

Next: Incident Response and Digital Forensics Basics.