Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 36 ยท Projects and Practice

Splunk Project: Analyze Web Access Logs from Ingest to Dashboard

A hands-on Splunk project: onboard web access logs, extract fields, write searches for traffic, errors and slow pages, build dashboard panels and an alert, and check the logic in Python.

In this project you take a realistic job from start to finish: a web team wants to know how busy the site is, where errors come from and which pages are slow. You will onboard the logs, write the searches, assemble a dashboard and add an alert. Splunk itself does not run in this page, so every SPL block is read-only, and each step has a runnable Python stand-in that computes the same answer from generated data so you can check your understanding of the logic.

The requirements

  1. Onboard nginx access logs into an index called web with sourcetype nginx:access.
  2. Show requests per hour, error rate, top pages, slowest pages and the busiest clients.
  3. Let viewers filter by status class and time range.
  4. Alert when the 5xx rate exceeds 5% over five minutes.

Step 1: onboarding

The log format (the nginx combined format plus the request time at the end):

Text
203.0.113.9 - - [02/Mar/2026:10:00:01 +0000] "GET /cart HTTP/1.1" 200 1532 "-" "Mozilla/5.0" 0.084

Add a monitor input on the web server's universal forwarder (see inputs, props and transforms):

INI
[monitor:///var/log/nginx/access.log]
index = web
sourcetype = nginx:access

And parsing on the indexer:

INI
[nginx:access]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = \[
TIME_FORMAT = %d/%b/%Y:%H:%M:%S %z
MAX_TIMESTAMP_LOOKAHEAD = 30
EXTRACT-nginx = ^(?<clientip>\S+) \S+ \S+ \[[^\]]+\] "(?<method>\S+) (?<uri_path>[^\s?]+)\S* \S+" (?<status>\d{3}) (?<bytes>\d+) "[^"]*" "(?<useragent>[^"]*)" (?<response_time>[\d.]+)
EVAL-status_class = substr(status,1,1)."xx"

Create the index (indexes.conf, 90 days, see indexes and retention) and confirm with index=web | head 10 that the fields clientip, method, uri_path, status, bytes and response_time appear in the field list.

Step 2: the searches

Traffic per hour:

SPL
index=web sourcetype=nginx:access
| timechart span=1h count AS requests

Error rate, as a single value and a trend:

SPL
index=web sourcetype=nginx:access
| timechart span=1h count AS total, count(eval(status>=500)) AS errors
| eval error_rate=round(errors/total*100,2)

Top pages and slowest pages (95th percentile response time):

SPL
index=web sourcetype=nginx:access
| stats count, perc95(response_time) AS p95, avg(response_time) AS avg by uri_path
| where count > 20
| sort - p95
| head 10

Busiest clients (possible bots):

SPL
index=web sourcetype=nginx:access
| stats count, dc(uri_path) AS pages, values(useragent) AS agents by clientip
| sort - count
| head 10

Try it: compute the same answers on generated logs (stand-in)

The stand-in generates 5,000 deterministic requests, then computes requests per hour, error rate, top pages and the 95th percentile by page.

Python Stand-in: stats over generated access logs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Splunk's perc95 is an approximation algorithm on large data, so exact values can differ slightly from this list-based version.

Step 3: the dashboard

Create a dashboard Web Operations (building dashboards) with:

  • A time input (default last 24 hours, token time_tok) and a dropdown status_class with values All (*), 2xx, 3xx, 4xx, 5xx (token class_tok).
  • A base search that keeps the needed fields: index=web sourcetype=nginx:access status_class=$class_tok$ | fields _time uri_path status response_time clientip useragent.
  • Panels as post-process searches: requests per hour (line), error rate (single value with trend), top pages (bar), slow pages (table), busiest clients (table with a drilldown setting $row.clientip$ that reveals a second panel listing that client's recent requests, see inputs, tokens and drilldowns).

A caution: filtering status_class in the base search makes the "error rate" panel useless when the filter is not All. Keep a separate base search for rate calculations, or calculate the rate from a base search that retains all classes.

Try it: the dropdown token and the base search (stand-in)

Python Stand-in: filter by status class

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 4: the alert

Create a scheduled alert that runs every 5 minutes over -5m@m to @m (alerts):

SPL
index=web sourcetype=nginx:access earliest=-5m@m latest=@m
| stats count AS total, count(eval(status>=500)) AS errors
| eval rate=round(errors/total*100,2)
| where total >= 50 AND rate > 5

The total >= 50 guard prevents a noisy alert at night when three requests happen and one of them fails. Trigger Once when the number of results is greater than 0, throttle for 30 minutes, and send an email with a link to the dashboard.

Try it: test the alert rule (stand-in)

Python Stand-in: five-minute error-rate alert

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Extensions

  1. Add a lookup of page names and owners and use it to show business names in the tables.
  2. Enrich clients with iplocation and show a map.
  3. Accelerate a Web data model over the data and move the panels to tstats.
  4. Save the dashboard, report and alert in a dedicated app and package it for your team (apps).

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. Why does the alert include total >= 50?

    A rate over a few requests is statistically meaningless; a minimum volume reduces false alarms.
  2. 2. Why should the error-rate panel not use a base search filtered by status class?

    The rate needs total requests and errors; if only one class is kept, the denominator is incomplete.
  3. 3. Which command computes the 95th percentile of response time per page?

    perc95 (or p95) is a stats aggregation; add by uri_path for one value per page.

Summary

  • You onboarded a log with inputs.conf and props.conf, and verified extracted fields.
  • Traffic, error rate, top and slow pages and busiest clients are simple timechart and stats searches.
  • A base search with post-process panels and tokenized inputs gives a fast, interactive dashboard.
  • A scheduled alert with a volume guard and throttle gives reliable notifications.

Next: Project: security monitoring dashboard