Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 36 ยท Fields and Extraction

Permanent Field Extractions: Field Extractor, props.conf, transforms.conf

Create permanent search-time field extractions with the Field Extractor, inline EXTRACT and REPORT rules in props.conf and transforms.conf, plus delimiters, sharing and testing.

A rex in a search disappears when you close it. A field extraction is saved with the source type so that everyone sees the field in every search, in dashboards and in data models. You can create one with the Field Extractor wizard or by writing configuration files. Both produce the same result.

The Field Extractor wizard

Open Settings > Fields > Field extractions > New is the plain form, but the friendly route is the wizard:

  1. Run a search that returns the events you want (for example index=app sourcetype=app_log).
  2. In the fields sidebar choose Extract New Fields, or open Settings > Fields > Field extractions and use Open Field Extractor.
  3. Pick a sample event and choose a method:
    • Regular expression: highlight the value in the sample, name the field and Splunk proposes a regex. Click Show Regular Expression to refine; add more samples to correct it. Use Validate to see non-matching events.
    • Delimiters: for CSV-like or key|value|key data; choose the delimiter (space, comma, tab, pipe) and name each column.
  4. Save. Set the permissions: This app only or All apps, and who can read it.

The wizard writes entries into props.conf (and transforms.conf for the delimiter method) in the app you chose, in the local folder.

What the configuration looks like

Inline extraction: EXTRACT

For one regex that belongs to one source type, put it directly in props.conf on the search head:

INI
# etc/apps/my_app/local/props.conf
[app_log]
EXTRACT-order = order (?<order_id>\d+)
EXTRACT-timing = took (?<duration_ms>\d+)ms

The name after EXTRACT- is a label (anything unique). Each regex uses named groups. You can also limit it to another field with ... in <field> at the end: EXTRACT-section = ^/(?<section>[^/]+) in uri_path.

Reusable extraction: REPORT with transforms.conf

If you want a named, reusable rule, or one with several options, define it in transforms.conf and call it from props.conf:

INI
# transforms.conf
[order_fields]
REGEX = order (?<order_id>\d+) .*?took (?<duration_ms>\d+)ms
INI
# props.conf
[app_log]
REPORT-orders = order_fields

For the delimiter method:

INI
# transforms.conf
[pipe_fields]
DELIMS = "|"
FIELDS = ts, user, action, status

REGEX with FORMAT handles dynamic pairs: REGEX = (\w+)=(\S+) plus FORMAT = $1::$2 creates a field named by each key.

Python Stand-in: applying props/transforms-style extraction rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Where do the files go?

Search-time extractions take effect on the instance that runs the search. In a distributed deployment that is the search head (and any apps you copy to the search peers if you use older replication of knowledge bundles). Index-time settings (line breaking, timestamps) go to the first full instance that parses the data. Do not put search-time extractions on a universal forwarder; they will never run there.

The knowledge bundle is how a search head ships its search-time knowledge to the indexers, so a search that filters on an extracted field can still be run in parallel on indexers.

Sharing and permissions

Knowledge objects have three scopes:

Scope Visible to
Private The creator only
App Everyone allowed to use that app
Global All apps

Change it with Settings > Fields > Field extractions > Permissions, or choose it in the wizard. Use app scope with roles unless the field is truly company-wide. A private extraction that you forget to share is the number-one reason "it works for me but not for my teammate".

Testing an extraction

After saving, search with the source type and check:

SPL
index=app sourcetype=app_log
| stats count(order_id) AS with_order, count(duration_ms) AS with_timing, count AS total

If the counts are lower than expected, display the events that miss the field with | where isnull(order_id) and adjust. Also check that the regex does not match more than intended: run | stats dc(order_id).

Python Stand-in: coverage check for an extraction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Performance and good practice

  • Anchor regexes and avoid leading .*; extraction runs on every event searched.
  • Prefer EXTRACT (inline) for simple one-off rules and REPORT plus transforms.conf for reusable or multi-regex rules.
  • Use the CIM names for fields where possible.
  • Keep extractions in an app under version control, not scattered in private objects.
  • For JSON and key=value logs, use KV_MODE = json or auto in props.conf instead of regex: [app_json] with KV_MODE = json.
  • Avoid index-time extraction unless you have a measured need.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where should a search-time field extraction be configured in a distributed deployment?

    Search-time knowledge applies where the search runs: the search head.
  2. 2. Which props.conf setting defines an inline regex extraction?

    EXTRACT-<name> holds a regex with named groups; REPORT-<name> points to a transforms.conf stanza.
  3. 3. Why might an extraction work for you but not for a colleague?

    Private knowledge objects are visible only to the creator until shared.
  4. 4. Which setting handles JSON events without a regex?

    KV_MODE controls search-time key-value extraction; json parses JSON automatically.

Summary

  • Use the Field Extractor (regex or delimiters) or write EXTRACT- and REPORT- settings in props.conf and transforms.conf.
  • Search-time extractions live on the search head; set permissions so others can use them.
  • Test coverage with count(field) versus count, and inspect the events that miss.
  • Use KV_MODE = json for JSON data; avoid index-time fields unless necessary.

Next: Lookups, field aliases and calculated fields