Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท Administration and Indexing

Splunk inputs.conf, props.conf and transforms.conf Explained

Learn the three core Splunk config files: inputs.conf to collect data, props.conf for line breaking and timestamps, transforms.conf for routing, filtering and masking data at index time.

Most of what Splunk does with data is controlled by three plain-text configuration files. inputs.conf says what to collect, props.conf says how to parse it, and transforms.conf holds the reusable rules that props refers to. Knowing them lets you fix the most common onboarding problems: events merged together, wrong timestamps, noisy data you do not want to pay for, and sensitive values that must be masked.

Where the data goes

Data passes through pipelines. On an indexer (or a heavy forwarder) the path is:

  1. Input: the data is read from a file, port, script or API and tagged with host, source, sourcetype and index.
  2. Parsing: Splunk breaks the stream into events (line breaking), finds timestamps, and applies index-time transforms (routing, filtering, masking).
  3. Indexing: events are written to buckets.

A universal forwarder usually does no parsing (apart from structured files such as CSV and JSON); the first full Splunk Enterprise instance in the path (a heavy forwarder or the indexer) parses. Therefore props.conf for line breaking belongs on the indexers or heavy forwarders, while search-time settings belong on the search heads.

inputs.conf

Each stanza defines one input:

INI
[monitor:///var/log/nginx/access.log]
index = web
sourcetype = nginx:access
disabled = 0

[monitor:///var/log/app/*.log]
index = app
sourcetype = app:log
blacklist = \.gz$
followTail = 0

[tcp://:5514]
index = network
sourcetype = syslog
connection_host = ip

[udp://514]
sourcetype = syslog

[script://$SPLUNK_HOME/etc/apps/myapp/bin/df.sh]
interval = 300
sourcetype = df
index = os

Input types: monitor (files and directories, remembering position with a checkpoint, fishbucket), tcp/udp (network ports), script (scheduled or long-running programs), WinEventLog, and HTTP Event Collector ([http] and [http://token-name]; see REST API, HEC and SDKs). host defaults to the machine name; source is the path; sourcetype should always be set explicitly because it selects the parsing rules in props.conf.

props.conf

Stanzas in props.conf are keyed by sourcetype, source::<path pattern> or host::<name>. The key settings for onboarding a text log (nicknamed "the great eight" by administrators) are:

INI
[app:log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2}T)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
MAX_TIMESTAMP_LOOKAHEAD = 32
TRUNCATE = 20000
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2}T)
  • SHOULD_LINEMERGE = false with a good LINE_BREAKER is far faster than line merging with BREAK_ONLY_BEFORE.
  • LINE_BREAKER is a regex whose first capture group (the text it removes) marks the break between events. Here a new event begins at a line starting with an ISO date, so stack traces stay in one event.
  • TIME_PREFIX is a regex that matches the text before the timestamp, TIME_FORMAT is a strptime pattern, and MAX_TIMESTAMP_LOOKAHEAD says how many characters after the prefix to look at.
  • TRUNCATE caps the characters per event (default 10000); a value of 0 means no limit, which is dangerous.
  • EVENT_BREAKER helps universal forwarders split big streams safely for load balancing.
  • TZ sets the time zone when the log lacks one.

Search-time settings in the same file: EXTRACT-name = regex (named groups become fields), REPORT-name = transform_stanza, FIELDALIAS-, EVAL-, LOOKUP-, and KV_MODE (auto, json, xml, none). Index-time settings: TRANSFORMS-name = stanza1, stanza2 and SEDCMD-name = s/regex/replacement/g.

Try it: line breaking and timestamps (stand-in)

This stand-in applies a LINE_BREAKER-style regex and a TIME_FORMAT to a text block, showing how stack traces stay in one event.

Python Stand-in: break events and read timestamps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

transforms.conf

transforms.conf holds named rules, each with a REGEX and an action, referenced from props.conf. The most common index-time uses:

Drop unwanted events (do not index and do not pay license for them) by sending them to the nullQueue:

INI
# props.conf
[nginx:access]
TRANSFORMS-drop_health = drop_healthchecks

# transforms.conf
[drop_healthchecks]
REGEX = GET /healthz
DEST_KEY = queue
FORMAT = nullQueue

Route events to another index based on content:

INI
[route_pci]
REGEX = cardholder
DEST_KEY = _MetaData:Index
FORMAT = pci

Mask sensitive data with SEDCMD in props.conf:

INI
[app:log]
SEDCMD-mask_card = s/\b(\d{4})\d{8}(\d{4})\b/\1XXXXXXXX\2/g

Search-time transforms (REPORT-) use REGEX with FORMAT = field1::$1 field2::$2, or define lookups (filename, match_type, max_matches). Index-time changes only affect new data and cannot be undone for events already indexed, so test on a small copy first.

Try it: filter and mask events (stand-in)

Python Stand-in: nullQueue and SEDCMD

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Configuration habits

  • Put changes in an app under local/, never edit files under etc/system/default.
  • After editing parsing files on an indexer, restart Splunk (or use debug/refresh for search-time changes). Forwarder and cluster changes are pushed from the deployment server or cluster manager.
  • Test with a small file and Settings > Add Data (the data preview applies props live), then check index=... | head.
  • Verify which settings win with splunk btool props list <sourcetype> --debug (monitoring and troubleshooting).
  • Prefer search-time field extraction over index-time: it is flexible and does not need re-indexing.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which file defines which files and ports Splunk collects?

    inputs.conf defines monitors, network ports, scripts and HEC inputs. outputs.conf is for forwarding.
  2. 2. What does DEST_KEY = queue with FORMAT = nullQueue do?

    Events sent to the nullQueue are not indexed and do not count against the license.
  3. 3. Where should line-breaking settings in props.conf normally be placed?

    Parsing happens at the indexer or heavy forwarder; universal forwarders mostly pass data through.
  4. 4. Why is TRUNCATE = 0 risky?

    The default limit protects against huge or malformed events.

Summary

  • inputs.conf collects data, props.conf parses it per sourcetype, transforms.conf holds reusable rules.
  • Set LINE_BREAKER, SHOULD_LINEMERGE, TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD and TRUNCATE for every new sourcetype.
  • Use nullQueue to drop noise, _MetaData:Index to route, and SEDCMD to mask.
  • Index-time changes affect only new data, so test first and edit config inside apps.

Next: Users, roles and knowledge objects