Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 36 ยท Data Models and Pivot

Splunk Data Models and Datasets: Structure Your Data

Understand Splunk data models: root event, search and transaction datasets, child datasets and constraints, field types, inheritance, and searching with from, datamodel and tstats.

Splunk stores events and extracts fields at search time. That is flexible, but it means every analyst must remember which index, sourcetype and fields to use. A data model captures that knowledge once. It is a hierarchical, searchable description of one subject area (for example web traffic or logins) that hides the SPL details. Analysts then build reports in Pivot without writing a search, and apps such as Enterprise Security rely on data models for their detections.

Datasets: the building blocks

A dataset is a collection of data you can search. Splunk uses the word for several things:

Dataset kind Where it comes from
Lookup dataset A lookup table definition
Data model dataset An object inside a data model
Table dataset A curated, flat table built in the Table Editor (no SPL needed)

Inside a data model there are four kinds of dataset:

  • Root event dataset: events selected by a constraint, a search filter such as index=web sourcetype=access_combined.
  • Root search dataset: results of a transforming or generating search (... | stats count by user). It cannot be extended with events constraints; it can be accelerated only if it is streaming.
  • Root transaction dataset: groups events into transactions by a field and time rules (a session, for example).
  • Child dataset: a subset of its parent, adding a constraint (for example status>=500) and optionally more fields.
flowchart TD
    W["Web (root event)<br/>constraint: index=web sourcetype=access_combined"] --> S["Server errors (child)<br/>constraint: status>=500"]
    W --> A["API calls (child)<br/>constraint: uri_path=/api/*"]
    S --> S5["Gateway errors (child)<br/>constraint: status=502 OR status=504"]

A child dataset sees all its parent's events that also match its own constraint, and inherits every field.

Fields in a data model

Each dataset has fields. When you add a field you pick its source:

  • Auto-extracted: fields Splunk already finds at search time (status, clientip).
  • Eval expression: computed with an eval, for example if(status>=500,"error","ok").
  • Lookup: a field returned from a lookup, such as country from an IP range lookup.
  • Regular expression: extracted from another field with a named group.
  • Geo IP: latitude, longitude, city and country from an address.

Every field has a type (string, number, boolean, IPv4, timestamp, object, child count), a flag that decides visibility (Selected, Optional, Hidden, Hidden and Required), and may be inherited or extracted in this dataset. Children inherit everything from the parent; to retire a parent field, hide it in the child. The built-in fields host, source, sourcetype and _time are always present.

Data model field names should be short and consistent across models (src, dest, user, action). Consistency is the whole point of the Common Information Model, which standardizes these names.

Creating a data model

  1. Go to Settings > Data models > New Data Model, enter a Title, an ID (cannot be changed later) and the app.
  2. Click Add Dataset > Root Event and enter a constraint, for example index=web sourcetype=access_combined.
  3. Add fields: Add Field > Auto-Extracted and tick the ones you want, or define an Eval Expression field.
  4. Add child datasets with Add Dataset > Child, each with its own constraint.
  5. Save, set permissions, then consider acceleration (next lessons).

Data models are stored as JSON in datamodels/<id>.json inside the app. Because a data model describes the shape of data, it is good practice to keep it in version control and ship it with the app that owns the data.

Searching a data model

There are three main ways to search a data model from SPL.

SPL
| from datamodel:"Web"."Server_errors"
| stats count by clientip

from reads a dataset and then you can apply any command. datamodel is an older generating command:

SPL
| datamodel Web Server_errors search
| stats count by Web.clientip

And tstats is the fast one, used against accelerated data:

SPL
| tstats count from datamodel=Web where Web.status>=500 by Web.uri_path

Note the Web. prefix in tstats and datamodel: field names are qualified by the dataset's root name. tstats works on indexed fields and accelerated summaries, so it is orders of magnitude faster than searching raw events, but it can only use fields that are in the data model.

Try it: data model inheritance (stand-in)

This stand-in builds the hierarchy above and applies constraints to events. Each child applies its parent's constraint and its own, exactly like a data model.

Python Stand-in: root dataset, child constraints, inherited fields

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last event (index os) never reaches any dataset, because the root constraint excludes it first.

Design guidelines

  • One data model per subject, not per source: a "Web" model can cover Apache, nginx and a load balancer when their fields are normalized.
  • Use specific constraints (index and sourcetype, or tags) so the model does not scan everything. Constraints written with tags (tag=web) are better for sharing, because they follow the CIM.
  • Keep the hierarchy shallow (two or three levels) and base children on real analysis categories.
  • Start with a few fields. Fields are cheap to add and expensive to remove once reports depend on them.
  • Name datasets after what they contain, not who uses them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What defines which events belong to a root event dataset?

    A root event dataset is defined by a constraint such as index=web sourcetype=access_combined.
  2. 2. What does a child dataset inherit from its parent?

    A child narrows its parent with a new constraint and still has all the parent's fields.
  3. 3. Which command searches an accelerated data model most quickly?

    tstats reads indexed data and acceleration summaries instead of raw events.
  4. 4. Which field source computes a value with an expression such as if(status>=500,"error","ok")?

    Eval expression fields hold the result of an eval formula evaluated at search time.

Summary

  • A data model is a hierarchy of datasets that describes one subject area.
  • Root datasets are events, searches or transactions; children add constraints and inherit fields.
  • Fields can be auto-extracted, eval, lookup, regex or Geo IP and have types and flags.
  • Search models with from datamodel:, datamodel or the fast tstats.
  • Keep models focused, with specific constraints and consistent field names.

Next: Pivot and accelerated reports