Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 37 ยท Log Collection and Normalization

Syslog, CEF, JSON and Other Log Formats

Read and parse the log formats a SIEM receives: RFC 5424 syslog, CEF, key-value, JSON and Apache access logs. Includes runnable Python parsers with deterministic output.

Logs arrive in many shapes. Before a SIEM can search or correlate them, each shape must be understood and converted. This lesson shows the formats you will meet most often and how to parse them.

Syslog

Syslog is the classic Unix logging protocol, also used by routers, firewalls and appliances. Two versions exist.

RFC 3164 (BSD style) has no year and no zone:

TEXT
<34>Nov 12 10:15:02 web1 sshd[2210]: Failed password for root from 203.0.113.9 port 51512 ssh2

RFC 5424 (modern) has an ISO timestamp, structured fields and a message ID:

TEXT
<165>1 2026-11-12T10:15:02.003Z web1 sshd 2210 AUTH001 [meta@1 env="prod"] Failed password for root

The leading <34> is the priority: facility * 8 + severity. Here 34 = 4 * 8 + 2, so facility 4 (auth) and severity 2 (critical).

Python Decode syslog priority and header

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Transport: UDP 514 is fast but can lose messages; TCP and TLS (typically port 6514) are reliable and encrypted. Prefer TLS.

CEF (Common Event Format)

CEF is a vendor-neutral format with a fixed header followed by key=value extensions:

TEXT
CEF:0|ExampleVendor|ExampleFW|1.0|100|Connection denied|5|src=203.0.113.9 dst=10.0.0.5 dpt=3389 proto=TCP act=deny

Header fields: version, vendor, product, product version, signature ID, name, severity. Then extensions. LEEF (used by IBM QRadar) is similar.

Python Parse a CEF event

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

(Real CEF escapes | and = with backslashes; production parsers must handle that.)

Key-value and JSON

Many tools write key=value pairs or JSON lines. They are easy to parse and self-describing.

TEXT
time=2026-11-12T10:15:02Z user=alice action=login result=failure src=203.0.113.9
{"time":"2026-11-12T10:15:02Z","user":"alice","action":"login","result":"failure","src":"203.0.113.9"}

JSON handles nesting (a process with a parent) and types (numbers stay numbers). It is the preferred format for new sources.

Web server access logs

The Apache/nginx "combined" format records requests:

TEXT
203.0.113.9 - - [12/Nov/2026:10:15:02 +0000] "GET /login.php?id=1%27 HTTP/1.1" 200 512 "-" "Mozilla/5.0"
Python Parse access logs and count status codes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Windows Event Log (preview)

Windows stores events as XML in .evtx files with an event ID, channel and many data fields. Chapter 3 continues with them in the next lesson.

Comparing formats

Format Structure Pros Cons
Syslog 3164 Loose text Universal No year/zone, hard to parse
Syslog 5424 Header + structured data Precise Less common in devices
CEF / LEEF Header + key=value Many vendors Escaping edge cases
Key-value Pairs Simple Quoting problems
JSON Nested typed Best for parsers Larger, field names vary

When you can choose, ask each source to log in JSON over TLS. You skip fragile regular expressions and keep field types intact.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In syslog, what does the <34> prefix encode?

    Priority = facility * 8 + severity.
  2. 2. Why is traditional (RFC 3164) syslog timestamp awkward?

    You must infer year and zone from other evidence.
  3. 3. Which transport is reliable and encrypted for syslog?

    TLS over TCP avoids loss and protects content in transit.
  4. 4. What are the first fields of a CEF header?

    The seven pipe-separated header fields come before the key=value extension.

Summary

  • Syslog (3164 and 5424), CEF, key-value, JSON and web access logs are the common shapes.
  • Priority values decode into facility and severity.
  • Parse structure with regular expressions or JSON, and watch for escaping and missing time zones.
  • Prefer JSON over TLS for new sources.

Next: Windows Event Logs for the SIEM