Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 37 ยท Getting Started

SIEM Architecture and Data Flow

Understand how a SIEM is built: agents, forwarders, parsing pipelines, storage tiers, search and alerting. See the data flow and simulate a pipeline in Python.

Every SIEM, whether it is a commercial product or an open-source stack, is built from the same few layers. Knowing the layers lets you reason about problems such as "why did the alert fire late?" or "why is search slow?" without memorizing one vendor's screens.

This lesson walks the path of one log line from the machine that wrote it to the analyst's queue.

The pipeline

flowchart LR
    A["Sources<br/>servers, firewalls, cloud"] --> B["Collectors<br/>agents and forwarders"]
    B --> C["Ingest pipeline<br/>parse, normalize, enrich"]
    C --> D["Storage<br/>hot, warm, cold"]
    C --> E["Real-time rules<br/>correlation"]
    D --> F["Search and dashboards"]
    E --> G["Alerts and cases"]
    F --> G

Data flows left to right: collection, processing, storage, then analysis.

1. Sources

Anything that writes events: operating systems, network devices, cloud audit logs, identity providers, applications, databases, endpoint tools. Each speaks its own format.

2. Collectors

Collection happens in two main ways:

  • Agent-based: software on the host reads local files or event logs and sends them on (examples: Wazuh agent, Elastic Agent, Splunk Universal Forwarder).
  • Agentless: devices push logs over the network (syslog) or the SIEM pulls them through an API (cloud audit logs, SaaS apps).

Collectors add buffering so that a short network outage does not lose events, and TLS so logs are not readable in transit.

3. Ingest pipeline

Raw text becomes structured fields. The pipeline parses (extract user, IP, action), normalizes (map to common field names), and enriches (add geolocation, asset owner, threat-intelligence match). You will study this in chapter 3.

4. Storage tiers

Tier Holds Speed Cost
Hot Last days or weeks Fast search High
Warm Last months Slower Medium
Cold / archive Years (compliance) Restore first Low

Retention periods come from laws, contracts and investigation needs. Many organizations keep 30-90 days searchable and a year or more archived.

Two engines read the data. Real-time rules evaluate events as they arrive. Scheduled searches run every few minutes over stored data. Analysts also search by hand during investigations.

6. Alerting and case management

Matches become alerts, go to a queue or ticketing system, and may start automated playbooks (chapter 7).

Python Simulate an ingest pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the dead-letter queue: lines that fail parsing are kept so you can fix the parser later. Silent drops are a classic cause of blind spots.

Scaling and reliability

  • Load balancing: several collectors share the incoming stream.
  • Message queues (such as Kafka) sit between collectors and the pipeline to absorb spikes.
  • Indexing clusters spread data over many nodes, with replicas for failure.
  • Health monitoring: alert when a source goes quiet. A firewall that stops sending logs is itself a finding.
Python Detect a silent log source

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Add a "no data" detection for every critical source on day one. Attackers who disable logging create a gap, and a gap alert catches it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the ingest pipeline do?

    The pipeline converts raw text into structured, consistently named, enriched events.
  2. 2. Which storage tier is searched fastest?

    Hot storage holds recent data on fast disks for quick search.
  3. 3. Why keep a dead-letter queue?

    Failed lines are evidence of parser gaps. Dropping them silently creates blind spots.
  4. 4. A server that normally logs every minute goes silent. What is the best response?

    Missing logs can mean a failed agent or an attacker disabling logging, so alert on silence.

Summary

  • A SIEM chains sources, collectors, an ingest pipeline, tiered storage, detection and alerting.
  • Agents and agentless methods both feed the pipeline; buffering and TLS protect the stream.
  • Parse, normalize and enrich at ingest; keep failures in a dead-letter queue.
  • Monitor the health of log sources; silence is a signal.

Next: The SIEM Landscape: Splunk, Elastic, Sentinel and Wazuh