Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 36 ยท Reports, Dashboards and Alerts

Splunk Alerts: Scheduled, Real-Time, Throttling and Actions

Create Splunk alerts that tell you when something happens: scheduled versus real-time, trigger conditions, throttling, severity and actions such as email, webhook and log events.

An alert is a saved search that runs on a schedule (or continuously) and does something when its results meet a condition. It answers "tell me when". Alerts are the difference between a Splunk deployment people look at and one that watches for them. They use the same scheduler as reports, so scheduling rules, windows and priority all apply.

Creating an alert

Run a search that returns what you want to be warned about, then click Save As > Alert:

Text
index=web sourcetype=access_combined status>=500
| stats count
| where count > 50

Here the search itself holds the threshold, so any result means "alert". That is the cleanest pattern: the search returns rows only when something is wrong. Then set:

  1. Title and description.
  2. Permissions: Private or Shared in App.
  3. Alert type: Scheduled or Real-time.
  4. Time range and schedule (for scheduled alerts, for example every 5 minutes over the last 5 minutes).
  5. Trigger condition: when to fire.
  6. Throttle (optional).
  7. Trigger actions.

Scheduled versus real-time

Type How it works Cost
Scheduled Runs on a cron schedule over a fixed time range Predictable; use this by default
Real-time (per-result) Triggers on every matching event as it arrives A permanently running search that holds a CPU core; limited quota
Real-time (rolling window) A continuous search over a sliding window (for example 5 minutes) with a trigger condition Also holds resources

Prefer scheduled alerts that run every minute or five minutes: they are almost as quick and much cheaper. Real-time searches are best reserved for few, high-value cases (and for Splunk Enterprise Security real-time correlations in particular).

A common mistake is a time range that does not match the schedule. A search that runs every 5 minutes over the last 15 minutes sees every event three times and fires repeatedly; one that runs every 15 minutes over the last 5 minutes leaves gaps. Match them (-5m@m to @m) and consider a small delay (for example -6m@m to -1m@m) when events arrive late.

Trigger conditions

Trigger alert when options for a scheduled alert:

  • Number of Results (is greater than, less than, equal to, ...).
  • Number of Hosts or Number of Sources.
  • Custom: a search expression evaluated against the result, such as search count > 50.

And Trigger: Once (one alert for the whole result set, with the results attached) or For each result (one alert per row, useful when each row is an entity such as a user or host).

Throttling

Without throttling, an alert that fires every five minutes for a long incident sends dozens of messages. Throttle tells Splunk to suppress further triggers for a period (for example 60 minutes). With For each result you also choose fields to suppress by, for example host, so that a problem on web01 is throttled but a new problem on web02 still alerts. Throttling is per alert and applies after the condition is met.

Try it: evaluating a throttled alert (stand-in)

This stand-in runs a scheduled alert every five minutes over fake counts, applies a "greater than" condition, and throttles per host for 30 minutes.

Python Stand-in: trigger condition and throttle

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Trigger actions

When an alert triggers, Splunk runs actions:

  • Add to Triggered Alerts: lists the alert in Activity > Triggered Alerts with a severity (Info, Low, Medium, High, Critical) that you pick.
  • Send email: to addresses, with subject, message, and the results as inline table, CSV or PDF. Token placeholders such as $name$, $result.host$ (a field of the first result) and $job.sid$ are available in the subject and body. A mail server must be configured under Settings > Server settings > Email settings.
  • Log event: write the alert to an index as an event, so you can search alert history.
  • Output results to a lookup: store results for later enrichment.
  • Webhook: HTTP POST of a JSON payload to a URL (chat tools, ticket systems). Splunk Enterprise limits webhook targets with an allow list.
  • Run a script: legacy; prefer a webhook or an app action.
  • Actions added by apps: PagerDuty, ServiceNow, Slack, and others.

Alerts live in savedsearches.conf too:

INI
[Web - too many 5xx]
search = index=web status>=500 | stats count | where count > 50
cron_schedule = */5 * * * *
dispatch.earliest_time = -5m@m
dispatch.latest_time = @m
enableSched = 1
counttype = number of events
relation = greater than
quantity = 0
alert.suppress = 1
alert.suppress.period = 30m
alert.severity = 4
action.email = 1
action.email.to = [email protected]
action.email.subject = [Splunk] $name$ fired

Try it: expanding message tokens (stand-in)

Python Stand-in: build an alert email subject and body

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Designing good alerts

  • Alert on symptoms people feel (errors, latency, failed logins), not on every anomaly.
  • Make the search return nothing when healthy; thresholds in SPL keep the alert definition self-explanatory.
  • Include context in the message: what, where, how bad, and a link to the investigation dashboard.
  • Review Triggered Alerts and the alert history monthly: if nobody acts on an alert, delete or tune it.
  • Test with a lower threshold and a private permission first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which alert type is usually preferred because it is cheaper?

    Scheduled alerts do not hold a search slot all the time, and a short schedule is nearly as fast as real time.
  2. 2. What does throttling do?

    Throttling prevents repeated notifications during the same incident; you can suppress by field values such as host.
  3. 3. An alert runs every 5 minutes over the last 15 minutes. What is the likely problem?

    Overlapping windows re-evaluate the same events. Align the time range with the schedule.
  4. 4. What does 'For each result' trigger mode do?

    Each row triggers its own actions, which suits entities like hosts or users; combine it with suppression by field.

Summary

  • An alert is a saved search plus a trigger condition, a throttle and actions.
  • Prefer scheduled alerts with a time range that matches the schedule.
  • Throttle (optionally by field) to avoid notification storms.
  • Actions include triggered alerts, email, log event, lookup output, webhooks and app actions.
  • Alert on symptoms, put the threshold in the SPL and review noisy alerts regularly.

Next: Data models and datasets