Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 37 ยท Logs

Datadog Log Pipelines: Grok Parser, Remappers and Processors

Turn raw log lines into searchable fields with Datadog log pipelines: filters, the Grok parser, date, status and service remappers, trace ID remapper, and other processors.

A raw log line is just text. To filter by status code or group by endpoint you need fields. Datadog's log pipelines process every log as it arrives, extracting attributes, fixing the timestamp and severity, and linking the log to a service and a trace. This lesson explains the pipeline model, the important processors, and the Grok syntax, then lets you practice with a stand-in parser. Datadog does not run on this site, so the stand-in is an approximation of Grok written in Python.

What a log looks like inside Datadog

After intake, a log is a JSON document made of:

  • Reserved attributes with special meaning: host, source, service, status, timestamp (shown as date), message, trace_id and span_id.
  • Tags attached by the Agent or you (env:prod, team:web).
  • Attributes, the parsed custom fields (http.status_code, duration, user.id). In search you write them with an @ prefix.

If your app already logs JSON, Datadog parses it automatically into attributes. Pipelines are most useful for plain-text logs and for cleaning up names.

Pipelines and processors

A pipeline is a filtered, ordered list of processors. Logs that match the pipeline's filter (for example source:nginx) go through its processors top to bottom. Pipelines run in order, and a log can pass through several. Integration pipelines are created automatically when you use a known source, and they are read-only (clone them to edit). Your own pipelines live under Logs > Configuration > Pipelines.

The main processors:

Processor Purpose
Grok Parser Extracts attributes from text with patterns
Date Remapper Sets the official timestamp from an attribute
Status Remapper Sets the log severity (status) from an attribute
Service Remapper Sets the service
Message Remapper Sets the displayed message
Trace ID Remapper Links the log to an APM trace
Attribute Remapper Renames or moves an attribute, to a standard name
URL Parser Splits a URL into host, path, query params
User-Agent Parser Extracts browser, OS, device
Category Processor Adds a category from filter rules (for example status code ranges)
Arithmetic Processor Computes a new numeric attribute
String Builder Combines attributes into a string
GeoIP Parser Adds country and city from an IP
Lookup Processor Adds values from a mapping table

Pipelines can be nested, and every processor adds work to each matching log, so keep pipelines short and their filters specific.

The Grok parser

A Grok rule has a name followed by a pattern made of literal text and %{matcher:attribute} pieces:

TEXT
access_rule %{ip:network.client.ip} - - \[%{date("dd/MMM/yyyy:HH:mm:ss Z"):timestamp}\] "%{word:http.method} %{notSpace:http.url} HTTP/%{number:http.version}" %{integer:http.status_code} %{integer:network.bytes_written}

Applied to this line:

TEXT
203.0.113.9 - - [12/Nov/2026:10:15:32 +0000] "GET /cart HTTP/1.1" 200 5123

it extracts the IP, timestamp, method, URL, version, status code and bytes as attributes. The matchers you will use most:

Matcher Matches
word letters, digits and underscore
notSpace any run without whitespace
integer a whole number (converted to a number)
number a decimal number
data anything, including spaces (lazy)
ip an IPv4 or IPv6 address
date("pattern") a date with the given format
regex("...") your own regular expression

You can define helper rules (reusable names) and write multiple rules; the first that matches wins. Use the Log Samples panel in the processor editor: paste a real log, and Datadog highlights whether the rule matched and what it extracted. That feedback loop is the right way to develop a parser. Escape special characters like [, ], (, ) with a backslash.

Reserved remappers

After parsing you usually fix the standard attributes:

  1. Date Remapper: Datadog uses date, timestamp, @timestamp and a few more by default; add the processor if your field has another name, otherwise the timestamp is the intake time.
  2. Status Remapper: maps level/severity/loglevel strings (error, warn, INFO, numeric syslog levels) to the status facet. Without it everything is info.
  3. Service Remapper: use when the log body carries the true service name.
  4. Trace ID Remapper: map your logged trace ID attribute (for example dd.trace_id) so the log shows in the trace view and traces show their logs. APM libraries can inject trace IDs into log records for you when log injection is on.

Practical workflow

  1. Look at raw logs in the Log Explorer. Is it JSON? If yes, you may need only remappers.
  2. Create a pipeline with a filter like service:storefront.
  3. Add a Grok Parser, paste samples, iterate until all variations match.
  4. Add Date, Status and Trace ID remappers.
  5. Add Category or Arithmetic processors to compute fields (for example duration_ms).
  6. Open a facet for any attribute you want to group by (see searching).
  7. Watch Pipeline Scanner and parse failures; unmatched logs are still stored, just unparsed.

Cost angle

Parsing is free of extra charge, but what you parse decides what you can later turn into log-based metrics and facets. Dropping fields you will not use, and excluding noisy logs, is done in indexes and the Agent (see indexes and archives).

Python Stand-in: a mini Grok parser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: status remapper and category processor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The stand-in covers a small slice of the real matcher list. Real Grok supports many more matchers, helper rules and filters (such as json, keyvalue, xml) that this simplified version leaves out.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the role of the Status Remapper?

    It maps level-like attributes to the status facet; otherwise logs default to info.
  2. 2. How are parsed fields written in a log search?

    Attributes use @; tags and reserved attributes are written without it.
  3. 3. Which tool lets you test a Grok rule against sample logs?

    Paste samples and the editor shows whether the rule matched and what it extracted.
  4. 4. Why use the Trace ID Remapper?

    With the trace ID mapped, you can jump between a trace and its logs.

Summary

  • Logs have reserved attributes, tags and parsed attributes (searched with @).
  • Pipelines apply ordered processors to logs that match a filter; integration pipelines handle known sources.
  • Grok rules extract fields with %{matcher:attribute}; test them with sample logs.
  • Remap date, status, service and trace ID so logs are consistent and linked.

Next: Searching and Analyzing Logs