Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 36 ยท Getting Started

Getting Data Into Splunk: Add Data, Source Types and Indexes

Upload a file, monitor a folder or listen on a port with the Add Data wizard, then set the source type, timestamp, line breaking, host and index, and verify the result.

An empty Splunk is not very useful. In this lesson you get your first data in with the Add Data wizard and learn the four labels every event receives: host, source, sourcetype and index. The wizard is the friendliest tool, and everything it does can also be written in configuration files, which you will meet in configuration files and precedence.

Ways to get data in

Method Use it when
Upload One-time file for learning or testing
Monitor A file, folder or Windows event log that keeps growing
Forward Data comes from other machines (universal forwarder)
TCP/UDP Syslog or network devices send to a port
HTTP Event Collector (HEC) Applications send JSON over HTTPS
Scripted inputs, modular inputs, apps APIs, databases, cloud services

The Add Data wizard

Open Settings > Add Data (or the Add Data tile on Home). Choose Upload, Monitor or Forward.

Upload a sample file

  1. Choose Upload and select a file such as an access log (access.log).
  2. Set Source Type: Splunk looks at the file and suggests a type (for web servers, access_combined). Use the preview to check that events are split correctly and the timestamps are right.
  3. Input Settings: pick the app context, host (name of the machine that produced the data) and index (default main; use a separate test index while learning).
  4. Review, then Submit. Click Start Searching to run source="access.log".

Monitor a file or folder

Choose Monitor > Files & Directories, enter the path (for example /var/log/nginx), and decide Continuously monitor or Index once. Splunk then follows the file like tail -f, remembering how far it has read, even after a restart.

The four labels (metadata)

Field Meaning Example
host Machine that produced the event web01
source File path, port or script the data came from /var/log/nginx/access.log
sourcetype The format of the data; controls parsing access_combined
index Where the event is stored web

sourcetype is the most important label. It chooses the rules for line breaking, timestamps and default field extractions. Choose names carefully and keep them consistent: dashboards and apps assume particular sourcetypes (the Common Information Model, for example, relies on them).

Python Stand-in: guessing a source type from the first line

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Timestamp and line breaking

During the Set Source Type step you can change how Splunk finds events. These settings live in props.conf:

  • Event breaks: Every line (one event per line), Regex (a pattern marks the start), or Auto. Multi-line data such as stack traces needs a start-of-event regex.
  • Timestamp: Auto, Current time (use arrival time) or Advanced with a TIME_FORMAT such as %Y-%m-%d %H:%M:%S, a TIME_PREFIX regex to say where the timestamp begins, and MAX_TIMESTAMP_LOOKAHEAD for how many characters to read.
  • Time zone: set it explicitly if the log lacks an offset; a wrong zone shifts every event.

Click Save As to store the settings as a new source type that you reuse. Always check the preview: if events are merged, or the timestamps are the arrival time, fix it before indexing, because indexed data is not re-parsed.

Python Stand-in: what a wrong TIME_FORMAT does

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same text gives two different dates, and the second is valid but wrong. That is why explicit time formats matter.

Choosing an index

Everything lands in main unless you say otherwise. Create a dedicated index for each kind of data or retention need (Settings > Indexes > New Index): different retention periods, different access rights and cleaner searches. Details come in indexes, buckets and retention.

Verify your data

Run these searches and compare against what you expect:

SPL
index=web | head 5
index=web | stats count by sourcetype, host, source
index=web earliest=-24h | timechart span=1h count

If numbers are off, check the time range, the timestamp settings and the index.

The HTTP Event Collector, briefly

HEC lets an application post JSON with a token. You create a token in Settings > Data inputs > HTTP Event Collector, then send:

Terminal
curl -k https://splunk.example.com:8088/services/collector/event \
  -H "Authorization: Splunk <your-token>" \
  -d '{"event": {"action": "login", "user": "ann"}, "sourcetype": "app_json", "index": "web"}'

HEC is ideal for applications and serverless code; it avoids installing a forwarder.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which metadata field controls how Splunk parses the format of data?

    The sourcetype chooses line breaking, timestamp and extraction rules.
  2. 2. Why should you check the preview before submitting an upload?

    Parsing happens at index time; fixing it later means deleting and reloading the data.
  3. 3. What does the monitor input do?

    A monitor input keeps track of the read position, like tail -f that survives restarts.
  4. 4. Which method is best for an application that sends JSON events directly?

    HEC accepts JSON over HTTPS with a token.

Summary

  • Use the Add Data wizard to upload, monitor or forward data.
  • Every event carries host, source, sourcetype and index.
  • Set sourcetype, line breaking and time format before indexing.
  • Use separate indexes and verify with a quick stats count by search.
  • HEC and forwarders are the production ways to send data.

Next: Splunk architecture components