Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 36 ยท Security and Apps

Splunk REST API, HTTP Event Collector and SDKs

Automate Splunk: send data with the HTTP Event Collector, create and read search jobs through the REST API on port 8089, use tokens, output modes and the Python SDK.

Splunk's browser interface is only one client. Everything it does goes through a REST API that you can call yourself, and applications can send data through the HTTP Event Collector (HEC) without installing a forwarder. This lesson explains both, shows the request shapes, and gives you stand-ins that build and check the payloads. The product itself cannot be reached from this page, so nothing here makes a network call.

HTTP Event Collector (HEC)

HEC accepts events over HTTP(S), which suits applications, serverless functions, containers and any system that can make a POST request. It listens on port 8088 by default.

Set it up in Settings > Data inputs > HTTP Event Collector:

  1. Global Settings: enable HEC, set the default index, sourcetype and the port; leave Enable SSL on.
  2. New Token: name, source, sourcetype, allowed indexes and default index, and optionally indexer acknowledgement. Splunk shows a token (a GUID); treat it like a password.

Send an event with the Authorization: Splunk <token> header:

Terminal
curl -k https://splunk.example.com:8088/services/collector/event \
  -H "Authorization: Splunk 11111111-2222-3333-4444-555555555555" \
  -d '{"time": 1772445600, "host": "web01", "source": "checkout",
       "sourcetype": "app:json", "index": "app",
       "event": {"level": "ERROR", "msg": "payment failed", "order_id": 8812}}'

A successful reply is {"text":"Success","code":0}. Key points:

  • The body is one or more JSON objects (they may be concatenated or in a list). event is required and may be a string or a JSON object; metadata fields time (epoch seconds, decimals allowed), host, source, sourcetype, index and fields (indexed extra fields) are optional.
  • /services/collector/raw takes unstructured text; set metadata with query parameters (?sourcetype=...&index=...) or a channel.
  • /services/collector/health checks that the collector is up.
  • The token's allowed indexes control where events may go; sending to another index returns an error.
  • With indexer acknowledgement enabled, each request needs a X-Splunk-Request-Channel header with a GUID, and the client later asks /services/collector/ack whether the events were indexed. Use it when you cannot afford to lose events.
  • Typical HTTP status codes: 200 success, 400 malformed data, 401/403 bad token, 503 server busy (retry with back-off).

Batch events, because HEC performance is dominated by request overhead: send hundreds or a few megabytes per request rather than one event per request.

Try it: build and validate HEC payloads (stand-in)

Python Stand-in: build a HEC batch and check it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The REST API

Splunk's management port 8089 (HTTPS) serves a REST API with endpoints under /services and /servicesNS/<user>/<app>. Almost every object has an endpoint: search/jobs, saved/searches, data/indexes, authentication/users, server/info, apps/local, licenser/licenses, data/inputs/monitor, and so on. Authenticate with basic auth (a user name and password) or, better, with an authentication token created in Settings > Tokens and sent as Authorization: Bearer <token> (token authentication has to be enabled, and the token's creator needs the edit_tokens_own capability).

By default responses are Atom XML; add output_mode=json (or csv, xml) for clean data.

Run a search and fetch results:

Terminal
# 1. create the job (the search must start with "search" or a pipe)
curl -k -u admin:password https://splunk.example.com:8089/services/search/jobs \
  -d search="search index=web status>=500 | stats count by uri_path" \
  -d earliest_time=-1h -d output_mode=json
# -> {"sid":"1772445600.123"}

# 2. check status
curl -k -u admin:password \
  "https://splunk.example.com:8089/services/search/jobs/1772445600.123?output_mode=json"

# 3. read results once dispatchState is DONE
curl -k -u admin:password \
  "https://splunk.example.com:8089/services/search/jobs/1772445600.123/results?output_mode=json&count=0"

A search job moves through states: QUEUED, PARSING, RUNNING, FINALIZING, DONE (or FAILED or paused). For quick, streamed results use /services/search/jobs/export (search, output_mode=csv), which returns rows as they are produced and creates no job to poll. The oneshot mode (exec_mode=oneshot) returns results in the same response for small searches. Poll politely (every second or two), and delete the job when done (DELETE /services/search/jobs/<sid>); jobs also expire after their time-to-live (default 10 minutes, longer if saved).

Other everyday calls: POST /services/saved/searches creates a report or alert; GET /services/server/info returns version and build; POST /services/data/indexes creates an index; and the SPL command | rest /services/server/info splunk_server=* reads the same endpoints from a search.

Try it: a search job state machine (stand-in)

Python Stand-in: poll a search job until it is done

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

SDKs and tools

Splunk publishes SDKs for Python (splunk-sdk, import splunklib), Java, JavaScript and C#, which wrap the REST API. A Python example:

Python
import splunklib.client as client
import splunklib.results as results

service = client.connect(host="splunk.example.com", port=8089, token="...", scheme="https")
stream = service.jobs.export("search index=web | head 5", output_mode="json")
for item in results.JSONResultsReader(stream):
    print(item)

Use SDKs for custom modular inputs, custom search commands (Python splunklib.searchcommands) and external dashboards. Other options: the splunk CLI (splunk search, splunk add oneshot), the Splunk Terraform provider, and OpenTelemetry collectors that export to HEC.

Security rules: always use HTTPS, give tokens the least privilege, rotate them, never commit tokens to Git, and restrict network access to 8088 and 8089.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which header authenticates an HEC request?

    HEC uses Authorization: Splunk <token>; the token is created in the HTTP Event Collector settings.
  2. 2. Which port is the Splunk management (REST API) port by default?

    8000 is Splunk Web, 8088 is HEC, 8089 is the management port, 9997 is forwarder receiving.
  3. 3. What does adding output_mode=json do?

    The default response format is XML; output_mode selects JSON, CSV or XML.
  4. 4. Why batch events in HEC requests?

    Sending hundreds of events per request greatly reduces overhead and load.

Summary

  • HEC (port 8088) ingests JSON or raw text over HTTPS with a Splunk token; batch events and use acknowledgement when loss is unacceptable.
  • The REST API (port 8089) exposes almost everything: create a search job, poll its state, read results, or export directly.
  • Use output_mode=json, token authentication and polite polling.
  • SDKs for Python, Java, JavaScript and C# wrap the API; secure tokens and ports.

Next: Project: web access log analysis