Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 36 ยท Fields and Extraction

Splunk Fields: Default Fields, Metadata and Field Discovery

Learn what fields are in Splunk: default fields such as host, source, sourcetype and _time, internal fields, automatic key-value discovery, the fields sidebar and field types.

Fields are the key to turning a pile of text into data you can count, filter and chart. A field is a name with a value attached to an event: status=404, clientip=203.0.113.7. This chapter shows where fields come from and how to make your own. This first lesson covers the fields every event has and how Splunk discovers the rest.

Default and internal fields

Every event carries these without any extraction:

Field Meaning
_time The event timestamp (epoch seconds internally)
_raw The original event text
host Machine that generated the data
source File path, port or script the event came from
sourcetype The data format label
index The index holding the event
splunk_server The indexer that stored it
linecount Number of lines in the event

Fields beginning with an underscore are internal: _time, _raw, _indextime, _cd (bucket and offset id), _bkt, _serial. They are not shown by default in the fields sidebar. Use fields or table to display them: | table _time, _indextime, _raw.

The first four (plus index) are the metadata fields, written at index time and stored in the index files, so searching on them is very fast. Prefer index=... sourcetype=... host=... in the first line of a search.

Search-time field discovery

Most fields are not stored; Splunk extracts them while you search. By default Splunk uses automatic key-value discovery: if the event contains key=value pairs (even key="quoted value"), Splunk creates fields for them. Many source types also have built-in extractions: for access_combined you get clientip, method, uri_path, status, bytes, referer and useragent. JSON and XML data are expanded automatically when the source type is configured for it.

This is schema on read in action: add a new key=value to your logs and the new field appears in searches immediately with no change to Splunk.

In Smart and Verbose modes the fields sidebar shows:

  • Selected fields: those you chose (host, source, sourcetype by default).
  • Interesting fields: fields that appear in at least 20% of the events.
  • All fields: a dialog listing every discovered field with its coverage percentage; check the boxes to move fields into Selected.

Click a field name to see its top 10 values with counts and percentages, and quick reports such as Top values by time and Average over time.

Python Stand-in: automatic key=value discovery and field coverage

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Using fields in searches

SPL
index=web status=404                    # filter with a field
index=web | stats count by status       # group by a field
index=web | fields host, status, uri_path | table host status uri_path
index=web | fields - _raw               # drop a field to save memory

Useful field commands:

  • fields includes (+) or excludes (-) fields.
  • fieldsummary shows statistics about every field (count, distinct count, numeric stats).
  • fieldformat changes how a field looks without changing its value.
  • rename, eval and rex (next lessons) create or change fields.
SPL
index=web | fieldsummary maxvals=3
index=web | stats sum(bytes) AS bytes | fieldformat bytes = tostring(bytes, "commas")

Field names and values

  • Field names are case-sensitive: Status and status are different fields.
  • Field values are case-insensitive in the search command, but case-sensitive in where and eval.
  • Names with spaces or special characters need single quotes in eval: 'user name'.
  • A field can have several values in one event (a multivalue field), for example [email protected] [email protected].
  • Whether a field is a number or a string depends on how you use it: where bytes > 100 converts text to a number when it can.
Python Stand-in: multivalue fields and field value case

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Index-time versus search-time fields

Fields can be indexed, that is, written into the index at index time. Metadata fields always are. You can add custom indexed fields in transforms.conf with WRITE_META, but this is rarely a good idea: it increases index size, cannot be changed for existing data and cannot be removed. Extract at search time unless you have measured a need (for example, fields that tstats queries on large volumes).

Making fields consistent

Different sources name the same thing differently: src, src_ip, clientip, source_address. Splunk's Common Information Model (CIM) defines standard names (for example src, dest, user, action) so searches and dashboards work across sources. You map your fields to the CIM with field aliases and calculated fields, covered in lookups, aliases and calculated fields.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which fields are stored in the index at index time?

    Metadata is written at index time; most other fields are extracted at search time.
  2. 2. What does the 'interesting fields' list in the sidebar show?

    Selected fields are your choice; interesting fields meet the 20% coverage threshold.
  3. 3. Are field names case-sensitive in Splunk?

    Status and status are different fields.
  4. 4. Why is adding custom indexed fields usually discouraged?

    Search-time extraction is flexible and applies to old data; indexed fields are fixed.

Summary

  • Every event has _time, _raw, host, source, sourcetype and index.
  • Other fields are discovered or extracted at search time (key=value, built-in extractions, JSON).
  • The fields sidebar shows selected, interesting and all fields; use fields, fieldsummary, fieldformat.
  • Names are case-sensitive; values are case-insensitive in search but not in where.
  • CIM names make data from different sources comparable.

Next: Extract fields with rex and regular expressions