Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 37 ยท Log Collection and Normalization

Which Logs to Collect and Why

Decide which log sources matter most for detection: identity, endpoint, network, cloud and application logs. Prioritize by attack path, volume and value with a runnable scorer.

A SIEM is only as good as its data. Collect too little and you are blind; collect everything and you drown in cost and noise. The skill is choosing sources by security value, starting from how attacks really unfold.

Think in attack paths

Most intrusions pass through a handful of stages. Make sure each stage leaves a trail somewhere you can read.

Stage Question Useful logs
Getting in Who authenticated and from where? Identity provider, VPN, email gateway, web server
Running code What executed? Endpoint process creation, script logging
Staying What changed to persist? Scheduled tasks, services, startup items, new accounts
Moving Where did they go next? Authentication on servers, SMB/RDP flows, firewall
Taking data What left the network? Proxy, DNS, cloud storage and firewall egress
Covering tracks Did logs vanish? Log-clear events, agent health

Priority tiers

Collect first

Identity Sign-ins, MFA, admin changes
Endpoint Process and security events
Cloud audit Who changed what
Edge Firewall, VPN, proxy

Highest value per gigabyte.

1 / 2

Source details

Identity (directory, SSO, cloud IdP): successful and failed sign-ins, MFA results, password resets, new or changed administrators, group membership changes. These are the single most valuable logs because attacks usually need an identity.

Endpoints: process creation with command lines, logins, service and task creation, PowerShell and script logging, EDR alerts.

Network edge: firewall allows and denies, VPN sessions, proxy requests with user and URL, DNS queries. Useful for beaconing, scanning and data theft.

Cloud: control-plane audit logs (who created a key, changed a policy, opened a bucket). Data-plane logs (each object read) are bigger and often optional.

Applications and databases: authentication, privilege changes, errors, large exports.

Security tools: alerts from EDR, IDS and email security, so you can correlate them.

Python Rank sources by value per gigabyte

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Low value-per-gigabyte does not mean "skip". Firewall logs are bulky but essential for network investigations; consider filtering the noisiest fields or keeping them in cheaper storage.

Log quality checklist

For every source, confirm:

  1. Completeness: all hosts of that type send logs.
  2. Timeliness: events arrive within minutes.
  3. Accurate time: clocks synchronized (NTP) and a time zone recorded.
  4. Enough detail: process command line, user, source address and outcome.
  5. Integrity: logs are shipped off the host quickly so attackers cannot rewrite them.
  6. Retention: long enough for investigations (attackers are often present for weeks).

Auditing: you must turn it on

Many systems do not log security detail by default. Windows needs audit policies and PowerShell logging; Linux benefits from auditd; cloud providers need audit trails enabled and sent to storage you protect. A source "connected" to the SIEM may still send almost nothing useful.

Python Audit log-source readiness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Logs hold personal data (usernames, IP addresses, sometimes URLs). Collect what you need, restrict who can read it, and respect retention limits set by law and policy. Never collect passwords or full message bodies unless there is a clear, lawful need.

Do not log secrets. Applications sometimes write passwords, tokens or card numbers into debug logs. Mask them at the source, or the SIEM becomes a treasure chest for attackers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which source usually gives the best detection value for its size?

    Attacks need identities, and sign-in logs are small and rich in signal.
  2. 2. Why ship logs off a host quickly?

    Local logs can be tampered with; a copy in the SIEM preserves evidence.
  3. 3. What does 'audit policy' control on Windows?

    Many valuable events are not recorded until the audit policy enables them.
  4. 4. Which should never be written into logs?

    Secrets in logs become a liability; mask them at the source.

Summary

  • Choose sources by attack path and value, starting with identity, endpoint, cloud audit and the network edge.
  • Check completeness, timeliness, clock accuracy, detail, integrity and retention for each source.
  • Auditing often must be enabled before logs are useful.
  • Respect privacy and never log secrets.

Next: Syslog, CEF and Common Log Formats