Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 36 ยท Architecture and Setup

Splunk Deployment Topologies and Capacity Sizing

Compare single-instance, distributed and clustered Splunk deployments and learn a simple method to estimate daily ingest, indexer count, storage and retention.

How big should a Splunk deployment be, and which layout should you pick? The answer depends on how much data you index per day, how many people search, how long you keep data and how much downtime you can accept. This lesson shows the standard layouts and a back-of-the-envelope method for sizing. Vendor sizing guides are the authority for real purchases; the numbers here are rules of thumb that you must validate.

Layout 1: single instance

One server does input, indexing and search. Best for labs, small teams and proofs of concept.

  • Easy to install, back up and understand.
  • Limited by one machine's CPU, memory and disk; no redundancy.
  • Typically up to a modest daily volume and a handful of users.

Layout 2: distributed (separate search head and indexers)

A dedicated search head queries one or more indexers, and forwarders feed the indexers.

  • Searches run in parallel on all indexers, so adding indexers scales both indexing and search.
  • Search and indexing no longer compete for the same disk and CPU.
  • Still no automatic recovery if an indexer fails (its data is unavailable until it returns).

Layout 3: clustered

Add indexer clustering (a cluster manager, peer nodes, a replication factor and a search factor) and optionally search head clustering.

  • Replication factor (RF): how many copies of each bucket exist. RF 3 survives two peer failures.
  • Search factor (SF): how many of those copies are searchable; searchable copies include the index files, so searches can use them immediately after a failure.
  • More disk (RF multiplies raw storage), but high availability and easier maintenance.
flowchart TD
    F["Forwarders<br/>(auto load-balance)"] --> P1["Peer 1"]
    F --> P2["Peer 2"]
    F --> P3["Peer 3"]
    CM["Cluster manager"] -.coordinates.- P1
    CM -.coordinates.- P2
    CM -.coordinates.- P3
    P1 <-->|"replicate buckets"| P2
    P2 <-->|"replicate buckets"| P3
    SH["Search head cluster"] --> P1
    SH --> P2
    SH --> P3

An indexer cluster with replication, queried by a search head cluster.

Multi-site and other options

Multisite clustering places peers in several sites (data centers or availability zones) and controls how many copies sit in each site, so a whole site can be lost. Other add-ons include a deployment server for forwarders, a license manager and a monitoring console. Splunk Cloud Platform hides these decisions from you.

Sizing: a worked method

Estimate four things:

  1. Daily ingest (GB/day): total raw data from all sources, measured or estimated from log volume.
  2. Retention: days you must keep searchable (for example 90).
  3. Storage: compressed size on disk. A common planning ratio is about half of the raw size: roughly 15% for the raw data (compressed journal) plus about 35% for index files, with the exact figures depending on your data. Use your measured ratio when you have one.
  4. Indexers: a typical rule of thumb for a dedicated indexer is on the order of a few hundred GB per day under a normal search load, but this varies widely with hardware, search workload and apps. Treat any number as a starting point and run a pilot.

Formula for total disk:

disk = daily_ingest ร— retention_days ร— 0.5 ร— replication_factor, plus headroom (add 20-30%).

Python Stand-in: storage and indexer estimate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output is an estimate: change ratio and gb_per_indexer to match measured values from a test. The point is the shape of the calculation.

Search load matters too

Indexing is only half the picture. These factors raise the number of indexers or search heads you need:

  • Many concurrent users or heavy dashboards that run every minute.
  • Scheduled searches and alerts (each runs on a CPU core while active).
  • Real-time searches (avoid them where possible).
  • Data models with accelerations (extra storage and CPU).
Python Stand-in: how many searches can run at once?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

(These are the default limits.conf settings; your version may differ.) Searches beyond the limit queue, and users see "waiting" jobs, which is the usual sign that you need more search capacity or fewer scheduled searches.

Choosing a layout: a quick guide

Situation Layout
Learning, home lab Single instance
Small team, one site, can tolerate downtime Distributed
Business-critical logging or SOC Indexer cluster + search head cluster
Disaster recovery across data centers Multisite cluster
Do not want to run servers Splunk Cloud Platform

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a replication factor of 3 mean in an indexer cluster?

    RF is the number of copies of each bucket; RF 3 tolerates two peer failures.
  2. 2. What is the difference between replication factor and search factor?

    All RF copies hold the data, but only SF copies have the index files needed to search immediately.
  3. 3. Using about 0.5 as the compression ratio, how much disk do 100 GB/day for 30 days need at RF 1 (before headroom)?

    100 x 30 x 0.5 = 1,500 GB.
  4. 4. Which layout lets you lose an entire data center without losing data?

    Multisite clusters keep copies of buckets in several sites.

Summary

  • Single instance for labs, distributed for scale, clustered for availability, multisite for disaster recovery.
  • RF is how many copies; SF is how many of those are searchable.
  • Disk is roughly daily ingest x retention x ratio x RF, plus headroom.
  • Search load (users, scheduled searches, accelerations) affects sizing as much as ingest.

Next: Universal forwarder setup