Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 36 ยท Projects and Practice

Splunk Project: License Usage and Storage Capacity Planning

Plan Splunk capacity: measure daily ingest with license_usage.log, find the biggest sources, reduce volume, estimate storage and indexer needs and build a capacity dashboard.

Splunk is licensed by how much data you index per day (or, in other models, by workload or compute), and storage and hardware grow with the same number. A deployment that is not measured eventually surprises its owner twice: with a license warning and with a full disk. In this project you measure ingest, find where it comes from, cut the waste and forecast storage. The product is not available on this page, so the SPL is read-only; the Python stand-ins do the arithmetic on sample numbers.

Step 1: measure daily ingest

Splunk records every indexed byte in _internal, in license_usage.log. Daily volume in GB, from the license manager:

SPL
index=_internal source=*license_usage.log type=Usage
| eval GB=b/1024/1024/1024
| timechart span=1d sum(GB) AS daily_GB

The same log has fields idx (index), st (sourcetype), h (host) and s (source); note that for performance Splunk may squash h and s when there are many distinct values. Use idx and st for reliable attribution. The Monitoring Console and Settings > Licensing > Usage report show the same data without SPL.

Compare with the daily limit (the license quota shown on the Licensing page). If ingest regularly exceeds 80% of the quota, plan action; if it exceeds 100% you get warnings (troubleshooting).

Try it: usage against the quota (stand-in)

Python Stand-in: days over quota and warning count

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: find the big sources

SPL
index=_internal source=*license_usage.log type=Usage earliest=-30d@d
| stats sum(b) AS bytes by idx, st
| eval GB=round(bytes/1024/1024/1024,1)
| sort - GB
| head 15

Almost always a few sourcetypes account for most volume (verbose debug logs, Windows event channels, firewall allow logs, DNS, flow data). Make a table of owners and ask whether each source is needed, at what detail and for how long.

Try it: top contributors and savings (stand-in)

Python Stand-in: which sources to trim

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 3: reduce volume the right way

In order of preference:

  1. Fix the source: turn off debug logs, log only what is useful, and use sampling for chatty systems.
  2. Filter at the edge on the universal forwarder with a blacklist on monitor inputs or Windows event whitelists/blacklists.
  3. Drop at parse time with a nullQueue transform (inputs, props and transforms); the event is not indexed and not licensed.
  4. Use a metrics index for numeric series instead of text events; it is cheaper to store and search.
  5. Summarize high-volume data into a summary index (scheduled stats/tstats results with collect) and keep raw data for a shorter period.
  6. Change retention only after deciding the compliance and investigation needs.

Never drop security-relevant data merely to save money without agreement from the data owner.

Step 4: estimate storage

On disk, indexed data is roughly half its raw size (about 15% for the compressed raw journal and 35% for index files), but the figure varies with the data (structured, repetitive logs compress well; many unique values, such as long hashes, compress badly). Measure your own ratio:

SPL
| dbinspect index=web
| stats sum(rawSize) AS raw_bytes, sum(sizeOnDiskMB) AS disk_mb
| eval ratio=round(disk_mb*1024*1024/raw_bytes,2)

Then storage = daily raw GB x ratio x retention days (x replication factor and search factor in a cluster, plus headroom of at least 20-30%). Accelerated data models and summary indexes add more.

Try it: storage forecast (stand-in)

Python Stand-in: disk for 90 days with a cluster

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 5: size the indexers

Indexer capacity depends on CPU, RAM, disk speed and search load, so published reference hardware and Splunk's sizing guidance are the authority. A commonly quoted planning figure is that a reference-spec indexer (about 12-16 modern cores, 32-64 GB RAM, fast local disk) handles on the order of a few hundred GB of indexing a day if it is not also serving heavy searches, and much less with many concurrent searches or premium apps such as Enterprise Security; always confirm against current Splunk documentation and test with your own data. Also size by searches: each concurrent search consumes about one core on the search head and on the indexers. A simple rule: peak ingest divided by per-indexer capacity, plus one or two for failure, plus more indexers for heavy search load.

Step 6: a capacity dashboard

Build Splunk - Capacity with: daily ingest vs quota (line plus a quota line using eval quota=100), top sourcetypes (bar), disk used per index (| rest /services/data/indexes fields currentDBSizeMB, maxTotalDataSizeMB), days of retention left per index, queue fill percentage and skipped-search counts. Add alerts at 80% of license quota and at 80% of disk size. Review monthly and forecast growth: a source growing 5% a month doubles in about 14 months.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. Which log records how much data was indexed for licensing?

    license_usage.log in the _internal index records bytes indexed with index, sourcetype, host and source.
  2. 2. Which is the best first action to cut a noisy debug log?

    Removing data at the source saves everything downstream: network, license, storage and search time.
  3. 3. In a cluster with replication factor 3, storage compared with one copy is

    Replication and search factors multiply the stored copies.

Summary

  • Measure ingest with license_usage.log and attribute it by index and sourcetype.
  • Reduce volume at the source first, then at the forwarder, then at parse time; use metrics indexes and summaries where suitable.
  • Estimate storage from a measured compression ratio, retention, replication and headroom.
  • Size indexers for both ingest and search load, and keep a capacity dashboard with alerts.

Next: Practice questions and drills