Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 36 ยท Data Models and Pivot

Splunk CIM and Data Model Acceleration Explained

Learn the Splunk Common Information Model: standard field names, tags and event types, mapping vendor logs, and how data model acceleration and tstats make searches fast.

Two firewalls from different vendors log the same fact with different words: one writes src_ip=, another SourceAddress=, a third client. If every dashboard has to know all of them, nothing is reusable. The Common Information Model (CIM) is Splunk's answer: a shared set of data models and field names that every source is mapped onto. Once data is CIM compliant, the same search, report or detection works on any vendor. Acceleration then makes those searches fast.

What the CIM contains

The CIM is a documented set of data models, delivered by the Splunk Common Information Model add-on. Each data model lists its datasets, required fields and the tags that place events in it. Frequently used models:

Data model Subject
Authentication Logins, logouts, failures
Web HTTP requests and responses
Network Traffic Flows and firewall connections
Endpoint Processes, services, file system and registry
Malware Detections by antivirus and sandboxes
Intrusion Detection IDS/IPS alerts
Network Resolution (DNS) DNS queries and answers
Email Mail flow
Change Configuration and account changes
Vulnerabilities Scanner findings
Performance, Inventory, Alerts, Certificates, Updates, Data Loss Prevention and others

CIM field names are standard: src, dest, user, action, app, signature, vendor_product, bytes_in, bytes_out, status. Values are normalized too; for the Authentication model action is one of success, failure or unknown.

Making data CIM compliant

Mapping a new source is a repeatable recipe, usually done in an add-on (see apps and add-ons):

  1. Get the data in with the right sourcetype and correct timestamps and line breaking (inputs and props).
  2. Extract fields at search time (EXTRACT- or REPORT- in props.conf).
  3. Create field aliases and calculated fields so the vendor names become CIM names: FIELDALIAS-src = SourceAddress AS src; EVAL-action = case(result=="OK","success", result=="BAD","failure", true(),"unknown").
  4. Define event types: named searches such as vendor_login = sourcetype=vendor:auth.
  5. Tag the event types with the tags the data model expects: authentication for login events, plus default or the model's other required tags.
  6. Validate with the CIM Setup page and Data Model Audit, or with | datamodel Authentication Authentication search | head.

Because data models use tags in their constraints (tag=authentication), adding a new source needs no change to the model, only a new tag on an event type.

Try it: normalizing vendor events (stand-in)

This stand-in maps two vendor formats to CIM-style fields with aliases and an eval-style action mapping.

Python Stand-in: FIELDALIAS and EVAL to CIM names

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Why acceleration exists

Searching raw events means reading compressed event data and extracting fields for every event. A search over 30 days of web logs can take minutes. With data model acceleration, Splunk builds summaries in the background: tsidx files containing the data model's fields for the matching events, stored next to the buckets (under datamodel_summary in each index path). Searches such as tstats and Pivot then read the compact summary and finish in seconds.

To enable it, open the data model and choose Edit > Edit Acceleration:

  • Accelerate: on.
  • Summary range: how far back to keep summaries, for example 7 days or 1 month. Larger ranges use more disk and build time. Searches older than the range fall back to raw events (or return nothing with summariesonly=true).
  • Advanced: Backfill Time, Max Concurrent (parallel build jobs), Max Time and a custom Schedule for the summary-building search.

Only event-based root datasets (and streaming root searches) accelerate; transactions and transforming searches do not. Accelerated models use disk roughly in proportion to the volume of events matched, so choose the constraint and the range carefully and enable acceleration only for models that are actually searched.

tstats reads the summaries:

SPL
| tstats summariesonly=true count from datamodel=Authentication
  where Authentication.action=failure by Authentication.src, Authentication.user

With summariesonly=true only accelerated data is returned (fast but possibly missing the newest events that have not yet been summarized); with false (the default) Splunk also searches raw events for the time not yet summarized.

The Settings > Data models page shows each model's acceleration status: percentage complete, size on disk, and the last summary update. Rebuild deletes and recreates summaries, and is needed after changing a constraint or a field that is accelerated.

Try it: scanning raw events versus a summary (stand-in)

Python Stand-in: why tstats is cheap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical advice

  • Accelerate only data models that dashboards, Pivot or detections use.
  • Keep the summary range as short as the questions need.
  • Use tstats ... where index=... sourcetype=... to narrow scans further; remember that tstats filters on indexed fields.
  • Watch disk use and the summary-building searches in the Monitoring Console (monitoring and troubleshooting).
  • Prefer tags in constraints so new sources join the model automatically.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the purpose of the CIM?

    CIM normalizes vendor data onto common models and fields so searches and apps work on any source.
  2. 2. How do you make a new vendor's login events appear in the Authentication data model?

    The model's constraint uses tags, so tagging a well-mapped event type brings the events into the model.
  3. 3. What does summariesonly=true do in tstats?

    It skips raw events for the not-yet-summarized time range, trading completeness for speed.
  4. 4. Which datasets can be accelerated?

    Transactions and transforming searches cannot be summarized in this way.

Summary

  • The CIM standardizes models, field names and tags so vendor data becomes interchangeable.
  • Make a source compliant with field aliases, calculated fields, event types and tags.
  • Data model acceleration builds tsidx summaries for fast tstats and Pivot.
  • Choose the summary range and accelerated models carefully because of disk and build cost.

Next: Indexes, buckets and retention