Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 36 ยท Administration and Indexing

Splunk Indexes, Buckets and Data Retention

Understand how Splunk stores data: indexes, event versus metrics indexes, the hot, warm, cold, frozen and thawed bucket lifecycle, indexes.conf settings and retention planning.

An index is where Splunk stores the data it receives. Choosing indexes well is one of the most important administrative decisions: indexes control who can search what (roles are granted per index), how long data is kept, where it lives on disk and how fast searches run. This lesson explains the storage model and the settings you will edit in indexes.conf.

What an index contains

Each index is a directory tree holding buckets. A bucket is a directory with:

  • the rawdata journal: the compressed original events, stored once;
  • tsidx files: time-series index files mapping terms and fields to positions in the raw data;
  • metadata files for host, source and sourcetype.

Splunk compresses data well; a rough rule of thumb is that raw data plus index files take about half of the original size on disk (about 15% for the compressed raw journal and about 35% for tsidx files), but this varies strongly with the data. Measure your own data before planning capacity (license and capacity project).

There are two kinds of index:

Type Stores Searched with
Events (default) Text events with fields extracted at search time search, stats, tstats
Metrics Numeric measurements (name, value, dimensions) in a compact form mstats, mcatalog

Metrics indexes are much cheaper for high-volume numeric series such as CPU or request latency.

Built-in indexes include main (the default for data with no index specified), _internal (Splunk's own logs), _audit (searches and user actions), _introspection (resource use), _telemetry and summary (the default destination for summary indexing). Names starting with an underscore are internal and not searched by default.

Why several indexes

Create an index when data differs in access, retention or volume:

  • Access control: roles are limited with srchIndexesAllowed, so HR and security data need their own indexes.
  • Retention: firewall logs for 30 days, audit logs for seven years: different indexes, different frozenTimePeriodInSecs.
  • Performance: searching index=web reads only that index's buckets; one huge main index slows everything.
  • Volume isolation: a noisy source filling an index does not evict data you need.

Do not create an index per host or per day; buckets are already time-based. A typical environment has tens of indexes, not thousands.

Create one in Settings > Indexes > New Index (name, paths, size, retention) or in indexes.conf (the preferred method on managed deployments, pushed from the cluster manager or deployment server).

The bucket lifecycle

Buckets move through stages as they age, each a directory with its own location:

flowchart LR
    H["Hot<br/>being written"] -->|roll| W["Warm<br/>read-only, same volume"]
    W -->|"age or size limits"| C["Cold<br/>cheaper storage"]
    C -->|"retention exceeded"| F["Frozen<br/>deleted or archived"]
    F -.->|"manual restore"| T["Thawed<br/>searchable again"]

A bucket is created hot, then rolled to warm, cold and frozen as data ages.

  • Hot: open for writing, searchable. Each index has a few hot buckets (maxHotBuckets, default 3).
  • Warm: closed and read-only, stored in homePath. A hot bucket rolls to warm when it reaches its size (maxDataSize, auto = about 750 MB, auto_high_volume = 10 GB on 64-bit systems), when its time span is too wide (maxHotSpanSecs), when it has been idle too long (maxHotIdleSecs), or when Splunk restarts.
  • Cold: warm buckets move to coldPath when the number of warm buckets exceeds maxWarmDBCount or the home path size limit is reached. Cold storage can sit on cheaper disks.
  • Frozen: when the newest event in a bucket is older than frozenTimePeriodInSecs, or the index exceeds maxTotalDataSizeMB, the bucket is frozen: by default deleted. To keep it, set coldToFrozenDir (copy the raw data to an archive directory) or coldToFrozenScript.
  • Thawed: you copy a frozen bucket into thawedPath, rebuild its index files (splunk rebuild <bucket>) and it becomes searchable again; thawed data is never frozen automatically.

Bucket directories are named db_<newestTime>_<oldestTime>_<id> (epoch seconds), for example db_1767225600_1767139200_42. The name makes it easy to see which time range a bucket covers. A frozen decision is made on the newest event in the bucket, so a bucket is kept until all its events are older than the retention period: retention in practice can be longer than you set.

A typical indexes.conf

INI
[web]
homePath   = $SPLUNK_DB/web/db
coldPath   = $SPLUNK_DB/web/colddb
thawedPath = $SPLUNK_DB/web/thaweddb
frozenTimePeriodInSecs = 7776000      # 90 days
maxTotalDataSizeMB = 500000           # 500 GB, whichever limit is hit first wins
maxHotBuckets = 6
coldToFrozenDir = /archive/splunk/web

[metrics_os]
datatype = metric
homePath   = $SPLUNK_DB/metrics_os/db
coldPath   = $SPLUNK_DB/metrics_os/colddb
thawedPath = $SPLUNK_DB/metrics_os/thaweddb
frozenTimePeriodInSecs = 2592000      # 30 days

Both retention rules apply: data is frozen when either the age limit or the size limit is exceeded. Set maxTotalDataSizeMB high enough that age is the real limit, otherwise you will lose data earlier than you promised. The default frozenTimePeriodInSecs is about six years (188697600 seconds), and the default maxTotalDataSizeMB is 500000.

On an indexer cluster repFactor = auto marks an index as replicated, and SmartStore moves warm buckets to remote object storage with a local cache; both are managed from the cluster manager and are beyond this lesson.

Try it: a bucket lifecycle simulator (stand-in)

The stand-in creates one bucket per day and applies retention rules to decide the state of each.

Python Stand-in: hot, warm, cold and frozen buckets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try it: estimating retention against disk size (stand-in)

Python Stand-in: which limit freezes data first?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Operational notes

  • Put homePath on fast disks (SSD/NVMe) and coldPath on larger, slower ones.
  • Keep indexes on volumes ([volume:hot], [volume:cold] with maxVolumeDataSizeMB) to enforce a total size across many indexes.
  • Never delete bucket directories by hand on a running indexer; use splunk clean eventdata -index <name> (offline) or the retention settings.
  • Check usage with | dbinspect index=web (buckets, states, sizes, time spans) and | rest /services/data/indexes.
  • Archive frozen data to cheap storage if regulations require it, and test the thaw procedure before you need it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens by default to a bucket that reaches the frozen stage?

    Frozen buckets are deleted unless you configure coldToFrozenDir or a coldToFrozenScript to archive them.
  2. 2. Which bucket stage is the only one open for writing?

    Hot buckets receive new events. They roll to warm when they are full, idle, too wide in time, or at restart.
  3. 3. When is a bucket frozen because of age?

    The decision uses the newest event, so all events in the bucket are past retention before it is frozen.
  4. 4. Why create separate indexes?

    Indexes are the unit of access control, retention and storage placement.

Summary

  • Indexes hold buckets of compressed raw data plus tsidx files; event and metrics indexes exist.
  • Buckets move hot, warm, cold, frozen; frozen is deleted unless archived; thawed restores archived data.
  • Retention is controlled by age (frozenTimePeriodInSecs) and size (maxTotalDataSizeMB); the first limit hit wins.
  • Separate indexes for different access, retention and volume needs.
  • Check buckets with dbinspect and plan disk using measured compression.

Next: inputs.conf, props.conf and transforms.conf