Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 37 ยท Log Collection and Normalization

Parsing and Normalization With a Common Schema

Turn different log formats into one common schema so one rule works on every source. Learn field naming, ECS and CIM ideas, and build a multi-source normalizer in Python.

A Linux server says Failed password for alice. A Windows server says EventID 4625, TargetUserName alice. A cloud provider says ConsoleLogin, errorMessage: Failed authentication. They all mean the same thing: a failed login. Normalization gives them one shape, so a single detection rule can cover all of them.

Why normalize

Without normalization you write one rule per vendor and maintain them forever. With it you write the rule once on a field such as event.outcome = failure, and new sources join automatically once their parser maps the fields.

Common schemas

Several public schemas define standard field names:

Schema Used by Example fields
Elastic Common Schema (ECS) Elastic Security source.ip, user.name, event.action, event.outcome
Splunk Common Information Model (CIM) Splunk src, user, action, app
Open Cybersecurity Schema Framework (OCSF) Multiple vendors src_endpoint.ip, actor.user.name
Microsoft ASIM Microsoft Sentinel SrcIpAddr, TargetUsername, EventResult
Google UDM Google SecOps principal.ip, target.user.userid

The names differ, but the idea is the same. This course uses ECS-like names such as source.ip and event.outcome.

The three pipeline steps

flowchart LR
    R["Raw event"] --> P["Parse<br/>extract values"]
    P --> M["Map<br/>to schema fields"]
    M --> V["Validate<br/>types and required fields"]
    V --> S["Store"]
    V -->|fails| D["Dead letter"]

Parse what the source says, map it to your names, then check the result.

Build a normalizer

Python Normalize Linux, Windows and cloud logins

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three different raw formats, one detection. That is the payoff.

Field naming rules

  • Consistent case and separator: source.ip everywhere, never SrcIP in one parser and src_ip in another.
  • Types matter: ports are integers, IPs are IP types, times are timestamps. Sorting and comparison break otherwise.
  • Keep the original: store the raw message (event.original) so you can re-parse later.
  • Controlled values: event.outcome is only success, failure or unknown.
  • Document the mapping in a table so detection engineers can trust it.

Grok and regular expressions

Many SIEMs let you write extraction patterns. Grok (Logstash, Elastic) uses named building blocks:

TEXT
%{SYSLOGTIMESTAMP:ts} %{HOSTNAME:host} sshd\[%{INT:pid}\]: Failed password for %{USER:user} from %{IP:src_ip}

Read-only example; the equivalent regular expression is below, and runnable.

Python Regex with named groups and type conversion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Test your parsers

Keep a folder of sample log lines per source, including odd ones (empty fields, extra spaces, long messages, other languages, different versions). Run every parser against them whenever it changes. A silent parser regression disables detections without any error.

Python A parser test table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second case fails on purpose: user names with spaces break the simple pattern. Finding such edge cases before production is exactly why parser tests exist.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main benefit of normalization?

    Common field names let a single rule cover every source that maps to them.
  2. 2. Which of these is an event schema?

    ECS defines standard field names for events; CIM, OCSF, ASIM and UDM are alternatives.
  3. 3. Why keep the original raw message?

    The raw text allows re-parsing and serves as evidence.
  4. 4. What should happen to events a parser cannot handle?

    Failures show gaps; dropping them hides blind spots.

Summary

  • Normalization maps each source's fields to a shared schema (ECS, CIM, OCSF, ASIM, UDM).
  • Pipeline: parse, map, validate; keep the raw event and a dead-letter path.
  • Use consistent names, correct types and controlled values.
  • Test parsers with a library of sample lines, including odd ones.

Next: Timestamps, Time Zones and Enrichment