Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 36 ยท Architecture and Setup

Universal Forwarder Setup: Install, Receive, Monitor and Verify

Install the Splunk universal forwarder, enable receiving on the indexer (port 9997), point the forwarder at it, add monitor inputs and verify the data arrives.

The universal forwarder (UF) is how most data reaches Splunk. It is a small, free agent that reads files and sends them to indexers. This lesson walks through the complete path: enable a receiver on the indexer, install the UF on a source machine, tell it where to send data, tell it what to read, and check that events arrive.

Step 1: enable receiving on the indexer

An indexer (or single instance) must listen for forwarder traffic.

In Splunk Web: Settings > Forwarding and receiving > Receive data > New Receiving Port, enter 9997, Save.

Or from the command line on the indexer:

Terminal
splunk enable listen 9997 -auth admin:<password>

Behind the scenes this writes to inputs.conf:

INI
[splunktcp://9997]
disabled = 0

Open the firewall for TCP 9997 between forwarders and the indexer.

Step 2: install the forwarder

Download the universal forwarder package for the source machine's operating system (it is a different download from Splunk Enterprise).

Linux:

Terminal
sudo useradd -m splunkfwd
sudo tar -xzf splunkforwarder-<version>-linux-amd64.tgz -C /opt
sudo chown -R splunkfwd:splunkfwd /opt/splunkforwarder
sudo -u splunkfwd /opt/splunkforwarder/bin/splunk start --accept-license
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunkfwd

Windows: run the MSI, accept the license, and in the wizard enter the receiving indexer (host and port 9997) and the data to collect (for example Windows Event Logs). Choose the service account (Local System can read all logs; a restricted account needs rights).

Step 3: point the forwarder at the indexer

Terminal
/opt/splunkforwarder/bin/splunk add forward-server idx1.example.com:9997

This creates outputs.conf in the forwarder's etc/system/local:

INI
[tcpout]
defaultGroup = primary_indexers

[tcpout:primary_indexers]
server = idx1.example.com:9997, idx2.example.com:9997
autoLBFrequency = 30
useACK = true
  • Listing several servers turns on automatic load balancing; the forwarder switches target every autoLBFrequency seconds.
  • useACK = true asks the indexer to confirm that data was written, so events are resent after a failure (at the cost of memory).
  • To encrypt traffic, add SSL settings (sslCertPath, sslPassword, sslVerifyServerCert) and use a receiving port configured for SSL.

Step 4: tell the forwarder what to read

Terminal
/opt/splunkforwarder/bin/splunk add monitor /var/log/nginx/access.log -index web -sourcetype access_combined

Which writes to inputs.conf:

INI
[monitor:///var/log/nginx/access.log]
index = web
sourcetype = access_combined
disabled = 0

Monitor stanzas accept wildcards: ... matches any depth (/var/log/.../*.log) and * matches within one path part. See configuration files and precedence for how stanzas are combined.

Python Stand-in: which files does a monitor path match?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Restart the forwarder after file changes if you edited the configuration by hand:

Terminal
/opt/splunkforwarder/bin/splunk restart

Step 5: verify

On the forwarder:

Terminal
/opt/splunkforwarder/bin/splunk list forward-server
/opt/splunkforwarder/bin/splunk list monitor

list forward-server shows each target as Active forwards or Configured but inactive forwards. If a target is inactive, check the network and the receiving port.

On the indexer, search the data and the forwarder's own logs:

SPL
index=web | stats count by host, source
index=_internal sourcetype=splunkd component=TcpInputProc | stats count by host

Forwarders also write their log to $SPLUNK_HOME/var/log/splunk/splunkd.log; connection errors appear there.

Python Stand-in: reading forwarder status output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Mass deployment

You rarely configure 400 forwarders by hand. Use a deployment server: define server classes (groups of forwarders matched by host name patterns) and attach apps (folders with inputs.conf and outputs.conf). Forwarders set deploymentclient.conf pointing to the deployment server and pull updates periodically. Another common approach is configuration-management tools (Ansible, Puppet, Chef) that copy the same app folders.

Common problems

Symptom Check
No data at all Receiving port enabled? Firewall? list forward-server active?
Data in the wrong index index = in the monitor stanza, and the index exists on the indexer
Only old data, nothing new File permissions, or the forwarder service account cannot read the folder
Duplicate events Two forwarders reading the same file, or rotated files read again; use crcSalt carefully

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which port do indexers commonly use to receive data from forwarders?

    9997 is the conventional splunktcp receiving port.
  2. 2. Which file tells a forwarder where to send data?

    outputs.conf holds the target indexers; inputs.conf says what to read.
  3. 3. What does useACK = true do?

    Indexer acknowledgment guards against data loss in transit.
  4. 4. How does a forwarder load-balance across several indexers?

    With several servers in a tcpout group, the forwarder switches targets automatically.

Summary

  • Enable a receiving port on the indexer, install the UF, set outputs.conf, add monitors in inputs.conf.
  • splunk add forward-server and splunk add monitor create the stanzas for you.
  • Verify with list forward-server and searches on index=_internal.
  • Use a deployment server or configuration management to roll out to many hosts.

Next: Configuration files and precedence