Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 36 ยท Architecture and Setup

Splunk Configuration Files: Stanzas, Layering and Precedence

Learn how Splunk configuration files work: stanzas, key-value settings, the etc folder layout, how settings merge across apps, btool and the order of precedence.

Splunk Web writes configuration files for you, but administrators read and edit them directly: they are the real source of truth, they can be copied between machines and kept in version control, and they are the only way to set many options. The tricky part is that the same file can exist in many places and Splunk merges them. This lesson explains the format and the rules of merging.

The file format

Configuration files end in .conf and contain stanzas (sections in square brackets) with key = value lines:

INI
# inputs.conf
[monitor:///var/log/nginx/access.log]
index = web
sourcetype = access_combined

[tcp://514]
sourcetype = syslog
connection_host = ip

Rules: # starts a comment (only at the start of a line), key names are case-sensitive, there are no quotes around values, and there is no trailing comment on a setting line.

The most important files

File Controls
inputs.conf What data to collect
outputs.conf Where a forwarder sends data
props.conf How data of a source, host or sourcetype is parsed and extracted
transforms.conf Reusable regular-expression and lookup rules referenced from props
indexes.conf Index definitions, paths, retention
savedsearches.conf Saved searches, reports and alerts
server.conf, web.conf Server and web settings
authentication.conf, authorize.conf Login methods and roles
limits.conf Search limits and resource controls

Each file has a documented specification (*.conf.spec) and example (*.conf.example) in $SPLUNK_HOME/etc/system/README.

Where files live

Settings live in $SPLUNK_HOME/etc in layers:

TEXT
etc/
  system/default/      <- Splunk's own defaults, NEVER edit
  system/local/        <- your global settings
  apps/<app>/default/  <- the app's shipped settings
  apps/<app>/local/    <- changes made in that app's context
  users/<user>/<app>/local/   <- a user's private objects

The rule of thumb: never edit anything under default/, because an upgrade overwrites it. Put your changes in local/, ideally inside your own app.

Precedence: who wins?

When the same setting appears in several files, Splunk applies a priority order. For global (index-time and system) settings, from highest to lowest:

  1. etc/system/local
  2. etc/apps/<app>/local (apps in reverse-ASCII order of name, so app zzz beats app aaa for the same setting)
  3. etc/apps/<app>/default
  4. etc/system/default

For search-time settings (extractions, saved searches, lookups) the app/user context matters first: the current user's private settings win, then the current app's local, then the app's default, then settings from other apps that are shared, then system. Whatever the context, files merge setting by setting (not whole file replaced): if system/default sets A and B, and system/local sets only A, you get your A and Splunk's B.

Python Stand-in: merging layered configuration

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

btool: see the merged result

btool prints the effective configuration and which file each line comes from. It is the first tool to reach for when a setting "does not work":

Terminal
splunk btool inputs list --debug
splunk btool props list access_combined --debug
splunk btool indexes list web

--debug prefixes every line with its file path. It reads the files from disk, so it also shows settings that have not yet been loaded; restart or reload when needed.

Applying changes

  • Many settings need a restart: splunk restart.
  • Some can be reloaded without restart: for search-time knowledge use the debug refresh (https://<host>:8000/en-US/debug/refresh) or the REST endpoint /services/.../_reload.
  • On a deployment server, splunk reload deploy-server pushes changes to clients.
  • Splunk Web notifies you when a restart is required.

Index time versus search time

A key concept for where a setting must live:

Index time Search time
When Event is parsed and written You run a search
Examples Line breaking, timestamp, host override, routing, masking Field extractions, lookups, calculated fields, tags
Where to configure First full Splunk instance (HF or indexer) Search head
Change effect Only new data Applies to old data too
Python Stand-in: a tiny .conf parser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Good habits

  • Package your changes as an app (etc/apps/my_company_config/local/...) so you can move them as a unit.
  • Keep config in Git and deploy with a deployment server or a configuration tool.
  • Use btool before you blame Splunk.
  • Never edit files in default.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where should you put your own configuration changes?

    Files under default are replaced on upgrade; local survives and has higher precedence.
  2. 2. What does the command splunk btool inputs list --debug show?

    btool applies the precedence rules and annotates the source file.
  3. 3. If system/default sets A=1 and B=2, and system/local sets A=9, what is the effective B?

    Merging is per setting: B is not overridden, so the default stays.
  4. 4. Which of these is an index-time setting?

    Line breaking and timestamps are applied when data is parsed and written.

Summary

  • Configuration is stanzas of key = value in .conf files, merged setting by setting across layers.
  • system/local beats app settings, which beat system/default; search-time precedence also depends on user and app context.
  • Edit local, never default; check results with splunk btool ... --debug.
  • Index-time settings belong on the parsing tier, search-time settings on the search head.

Next: Search basics and time ranges