Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 36 ยท Fields and Extraction

Extract Fields With rex, erex and regex in SPL

Extract new fields at search time with rex and named capture groups, mode=sed, max_match, plus erex, regex filtering, spath for JSON, extract and kvform. With examples.

When Splunk does not extract a field you need, you can pull it out of _raw yourself with a regular expression. The rex command does this at search time, which makes it perfect for exploring: you can try a pattern, see the result, and refine, without touching any configuration. When the pattern works and is needed by many people you turn it into a permanent extraction (next lesson).

rex with named groups

rex uses PCRE regular expressions. Each named capture group (?<name>...) becomes a field:

SPL
index=app "Card declined"
| rex "order (?<order_id>\d+)"
| stats count by order_id

The event text Card declined for order 8841 creates order_id=8841. Key points:

  • By default rex runs on _raw. Use field= to run on another field: rex field=uri "^/(?<section>[^/]+)".
  • Names must use letters, digits and underscores, and cannot start with a digit.
  • If the regex does not match, the field is simply not created for that event.
  • Wrap the pattern in double quotes and escape a literal double quote inside as \".

More examples:

SPL
| rex "client=(?<ip>\d{1,3}(?:\.\d{1,3}){3})"
| rex field=uri_query "(?:^|&)q=(?<search_term>[^&]+)"
| rex "took (?<ms>\d+)ms"
| rex "^(?<month>\w{3})\s+(?<day>\d+)\s+(?<time>\S+)\s+(?<host_name>\S+)\s+(?<process>[^\[:]+)"
Python Stand-in: rex named groups in Python (same regex ideas)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python writes named groups as (?P<name>...); Splunk's rex writes (?<name>...). The pattern logic is the same.

Greedy, lazy and anchors

Regex is easy to get wrong in a way that still "works" on the first example. Keep these rules in mind:

  • .* is greedy: it takes as much as possible. Prefer [^"]* (anything except a quote) or the lazy .*?.
  • Anchor with ^, $ or a stable literal so the pattern cannot match in a wrong place.
  • \d+ digits, \w+ word characters, \S+ non-space, [^,]+ anything until a comma.
  • Optional parts: (?:...)?. Alternatives: (?:GET|POST).
Python Stand-in: greedy versus precise patterns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

max_match: several values per event

By default rex takes the first match. max_match=0 returns all matches as a multivalue field:

SPL
| rex max_match=0 "(?<ip>\d{1,3}(?:\.\d{1,3}){3})"
| mvexpand ip

The default max_match is 1. Use max_match=0 for "unlimited".

rex mode=sed: replace text

mode=sed rewrites a field using a sed-style expression (useful to mask data in results):

SPL
| rex mode=sed field=_raw "s/\d{12,16}/XXXX/g"
| rex mode=sed "s/password=[^ ]+/password=****/g"

This only changes the search result, not the stored data. To mask data in the index, use SEDCMD in props.conf (index time).

regex (filtering), erex, spath

  • regex keeps or drops events by regex: | regex _raw="error\s+code\s+\d{3}", or | regex uri!="^/health" to exclude.
  • erex generates a regex from examples: | erex order_id examples="8841, 8842". Handy for exploration; it shows the regex it generated, which you can copy into rex.
  • spath extracts from JSON or XML: | spath input=_raw path=user.name output=user_name. With index=app sourcetype=json, fields are often auto-extracted; spath helps with nested fields embedded in a string.
  • extract (alias kv) re-runs key-value extraction on a field; kvform applies a form template; multikv extracts columns from table-like events (for output of top or ps).
SPL
index=api
| spath input=payload path=items{}.sku output=sku
| mvexpand sku

items{}.sku is Splunk's path syntax for arrays: braces select all elements.

Python Stand-in: spath-style extraction from JSON

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Debugging tips

  1. Test with | head 20 to see many sample events.
  2. Count the events that matched: | stats count(order_id) AS matched, count AS total.
  3. Show the unmatched: | where isnull(order_id) | head 10.
  4. Build the regex from a very small match and grow it.
  5. Remember rex scans text; run it after filtering to as few events as possible, because regex on millions of events costs CPU.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does rex name a new field?

    Each named group in the regular expression becomes a field.
  2. 2. What does max_match=0 do?

    The default is 1; 0 means unlimited matches.
  3. 3. Which command changes the display text in results to mask a password, without altering indexed data?

    rex mode=sed edits results at search time; SEDCMD in props.conf masks at index time.
  4. 4. Why is user="(.*)" a risky pattern?

    Use [^"]* or a lazy .*? to stop at the first closing quote.

Summary

  • rex "(?<name>pattern)" extracts fields from _raw or from field= at search time.
  • Prefer precise patterns ([^"]*) and anchors over greedy .*.
  • max_match=0 collects every match; mode=sed rewrites text in results.
  • regex filters events, erex suggests a regex, spath reads JSON and XML.
  • When a rex is stable and shared, make it a permanent extraction.

Next: Permanent field extractions with props.conf and transforms.conf