Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 36 ยท Administration and Indexing

Monitoring and Troubleshooting Splunk: _internal, Queues and the Monitoring Console

Find out what is wrong with Splunk itself: the Monitoring Console, the _internal index, blocked queues, skipped searches, license usage, the search job inspector and the CLI helpers.

Splunk monitors your systems, but who monitors Splunk? The answer is also Splunk: every component writes logs and metrics to internal indexes, and the Monitoring Console turns them into dashboards. Most day-to-day problems (data not arriving, slow searches, skipped schedules, license warnings) can be diagnosed in a few searches and a few CLI commands. This lesson gives you the map.

The Monitoring Console

In Splunk Web open Settings > Monitoring Console (MC). It runs on one instance (often a search head or the license manager), in standalone mode for a single server or distributed mode where you add the other instances as search peers and assign them roles. Key dashboards:

  • Overview: instance health and resource usage.
  • Indexing > Performance, Indexes and Volumes: indexing rate, queue fill percentage, and disk use per index.
  • Search > Activity, Scheduler activity: running and skipped searches, concurrency, longest searches.
  • Resource Usage: CPU, memory, I/O per instance.
  • Forwarders: deployment of forwarder connections and missing forwarders.
  • Health Check: a set of checks you can run on demand, for example disk free, search peer reachability and version mismatches.

The MC can send platform alerts (for example "Near Critical Disk Usage", "Missing forwarders") that you enable under Monitoring Console > Settings > Platform Alerts.

The internal indexes

Index What it holds
_internal Splunk's own logs: splunkd.log, metrics.log, scheduler.log, web_access.log, license_usage.log
_audit Who ran which search and who logged in or changed settings
_introspection Resource usage (CPU, memory, disk) of processes and hosts
_telemetry Anonymous usage data, when enabled
_configtracker Changes to configuration files (newer versions)

Internal indexes are not part of normal searches; address them explicitly, and limit the time range. Useful starting searches:

SPL
index=_internal source=*splunkd.log log_level=ERROR OR log_level=WARN
| stats count by component
| sort - count
SPL
index=_internal source=*scheduler.log status=skipped
| stats count by savedsearch_name, reason
SPL
index=_internal source=*license_usage.log type=Usage
| timechart span=1d sum(b) AS bytes
| eval GB=round(bytes/1024/1024/1024,2)
SPL
index=_internal source=*metrics.log group=queue
| eval fill=round(current_size_kb/max_size_kb*100,0)
| timechart span=5m max(fill) by name

The metrics.log events with group=queue show every pipeline queue (parsingqueue, aggqueue, typingqueue, indexqueue): a queue that stays near 100% means the stage after it cannot keep up. Blocked queues are logged in splunkd.log too (blocked=true). group=per_sourcetype_thruput shows indexing volume per sourcetype.

Try it: spotting blocked queues (stand-in)

This stand-in parses metrics.log-style lines, computes the fill percentage and flags queues that are almost always full.

Python Stand-in: queue fill from metrics.log

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common problems and what to check

Data is missing. Walk the path from the source forward:

  1. Is the forwarder running and connected? splunk list forward-server and index=_internal host=<forwarder> source=*splunkd.log.
  2. Is the input being read? splunk list monitor, splunk list inputstatus, and search index=_internal source=*splunkd.log component=TailingProcessor. Permissions on the file are a frequent cause.
  3. Is the index name valid? An event sent to an unknown index is dropped, and splunkd.log says so.
  4. Are timestamps wrong? Events dated in the future or the past are indexed but not found in your time range. Search index=* earliest=-1y latest=+1y | head.
  5. Is the indexer blocked? Check queues and disk free (an indexer stops indexing when free space falls below minFreeSpace, default 5000 MB).

Searches are slow. Open the Job Inspector (Job > Inspect Job) of a finished search: it shows the execution costs per component and phase and how many events were scanned versus returned. If many events are scanned and few returned, add index, sourcetype, a narrower time range or use tstats. Look at the search.log link for details. Avoid leading wildcards (*error), join, and transaction on large data. | tstats, summary indexes and accelerated data models are the fixes at scale.

Scheduled searches are skipped. The scheduler.log explains with a reason such as "The maximum number of concurrent historical searches on this instance has been reached". Spread schedules, widen windows, raise priority for important ones, or add search head capacity.

License warnings. A volume license limits how much new data is indexed per day. Exceeding it creates a warning; too many warnings in a rolling window block searching (the exact thresholds depend on license type and size; check the current license terms). Find which sourcetypes or indexes grew with the license_usage.log search above split by idx or st, then drop noise with a nullQueue transform.

Command line helpers

All commands live in $SPLUNK_HOME/bin:

Command Use
splunk status / restart State and control of the service
splunk btool <conf> list --debug Show merged settings and which file each came from
splunk list monitor Files and directories being monitored
splunk list inputstatus Read position per input
splunk cmd splunkd print-modinput-config Debug modular inputs
splunk validate files Check configuration files for errors
splunk diag Collect a diagnostic archive for Splunk Support
splunk show config Show running settings of the instance

Logs are in $SPLUNK_HOME/var/log/splunk (splunkd.log, metrics.log, scheduler.log, web_service.log), so you can read them even when searching is broken.

Try it: a troubleshooting decision helper (stand-in)

Python Stand-in: from symptom to first checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where do you look for Splunk's own scheduler and license logs?

    _internal holds scheduler.log, license_usage.log, splunkd.log and metrics.log.
  2. 2. What does a queue that stays near 100% in metrics.log suggest?

    A full queue means downstream processing (often disk I/O or indexing) is the bottleneck.
  3. 3. Which tool shows how a slow search spent its time?

    The Job Inspector lists execution costs per phase and the events scanned versus returned.
  4. 4. A new log source is indexed but you cannot find its events in the last hour. What is a likely cause?

    Wrong time zone or format leads to events with the wrong _time; widen the time range and check _time.

Summary

  • The Monitoring Console and the internal indexes (_internal, _audit, _introspection) show Splunk's own health.
  • Searches over metrics.log, scheduler.log and license_usage.log answer most operational questions.
  • Missing data: walk source, forwarder, input, index, timestamp and queues.
  • Slow searches: Job Inspector, narrower filters, tstats.
  • Learn the CLI helpers: btool, list monitor, validate files, diag.

Next: Splunk for security detections