Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 43 ยท Projects

Project: Log Search and Dashboards

Build log search and a dashboard: ingest and parse logs, search by level, count by time with a histogram, compute error rate and top-N, with a Python stand-in.

Logs are the other great Elasticsearch use case. Servers write millions of lines such as GET /checkout 500 812ms, and when something breaks you need to answer questions quickly: how many errors in the last hour, which endpoint fails most, when did it start? In this project you design the pipeline and build the same calculations in Python, using generated logs.

The Elasticsearch requests are read-only, because no cluster can run on this site. The Python program runs in your browser. It does by hand the group-by work that Elasticsearch aggregations do at scale, so you can see exactly what each dashboard number means.

The pipeline

From log line to dashboard

Collect An agent ships raw lines.
Parse Split lines into typed fields.
Index Store in a data stream by day.
Aggregate Charts ask for counts and ratios.

Parse once at ingest, then every query is cheap.

1 / 2

Typical tools: a shipper such as Filebeat or Elastic Agent collects the lines, an ingest pipeline (or Logstash) parses them, and Kibana draws the dashboards. The part you control the most is the data model.

Step 1: parse into fields

A raw line is only text. To filter and aggregate you need fields. An ingest pipeline with a grok or dissect processor splits the line, and a date processor turns the timestamp text into the real @timestamp:

HTTP
PUT /_ingest/pipeline/web-logs
{
  "processors": [
    { "dissect": {
        "field": "message",
        "pattern": "%{ts} %{level} %{service} %{method} %{path} %{status} %{duration_ms}ms"
    } },
    { "date":    { "field": "ts", "formats": ["ISO8601"], "target_field": "@timestamp" } },
    { "convert": { "field": "status",      "type": "integer" } },
    { "convert": { "field": "duration_ms", "type": "integer" } },
    { "remove":  { "field": "ts" } }
  ]
}

Parsing at ingest time is the right trade-off: it costs a little CPU once per line, and every later query works on typed fields instead of scanning text. Lines that fail to parse should not vanish; send them to a failure index with on_failure, so you notice changes in the log format.

Step 2: mapping and data streams

Logs are append-only and time-based, so use a data stream: a named target that creates hidden, time-ordered backing indexes and rolls over automatically (geo and time-series data). Map it with an index template:

HTTP
PUT /_index_template/web-logs
{
  "index_patterns": ["logs-web-*"],
  "data_stream": {},
  "template": {
    "settings": { "index.default_pipeline": "web-logs" },
    "mappings": {
      "properties": {
        "@timestamp":  { "type": "date" },
        "level":       { "type": "keyword" },
        "service":     { "type": "keyword" },
        "method":      { "type": "keyword" },
        "path":        { "type": "keyword" },
        "status":      { "type": "short" },
        "duration_ms": { "type": "integer" },
        "message":     { "type": "text" }
      }
    }
  }
}

Everything you group or filter by is a keyword or a number. Only the original message is text, for free-text search. An index lifecycle policy then moves old data to cheaper storage and deletes it after the retention period (for example 30 days).

Step 3: the queries

Search recent errors for one service (all exact conditions, so filters):

HTTP
GET /logs-web-*/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term":  { "level": "ERROR" } },
        { "term":  { "service": "checkout" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ]
    }
  },
  "sort": [ { "@timestamp": "desc" } ],
  "size": 20
}

Dashboard numbers in one request, with no hits returned:

HTTP
GET /logs-web-*/_search
{
  "size": 0,
  "query": { "range": { "@timestamp": { "gte": "now-6h" } } },
  "aggs": {
    "over_time": {
      "date_histogram": { "field": "@timestamp", "fixed_interval": "10m" },
      "aggs": {
        "errors": { "filter": { "range": { "status": { "gte": 500 } } } }
      }
    },
    "top_paths": { "terms": { "field": "path", "size": 3 } },
    "latency": { "percentiles": { "field": "duration_ms", "percents": [50, 95, 99] } }
  }
}
  • date_histogram buckets documents by time. The nested filter aggregation counts the 5xx responses in each bucket, so you can compute an error rate per bucket (sub-aggregations).
  • terms gives the top-N values by count, and percentiles give latency distribution. Averages hide slow outliers, so dashboards usually show p95 or p99.
  • Percentiles and terms counts are approximate on big data; this is the cost of speed (aggregations basics).

Step 4: the stand-in

The Python program below generates 600 deterministic log lines (a simple pseudo-random generator, so the output never changes), parses them with a regular expression like the ingest pipeline does, and then computes the dashboard: a histogram per 10 minutes, the error rate, top paths and percentile latency. A burst of 5xx errors on /checkout is planted at 10:30-10:40.

Python Generate, parse and aggregate logs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the histogram: most buckets show a low error rate, and the 10:30 bucket spikes, which is exactly the planted incident. The "top error paths" list names /checkout as the culprit. This is the investigation flow of a real dashboard: when (histogram), where (top-N), how bad (rate and percentiles). Then you search the individual lines from that time window to find the cause.

Notice also that the mean latency is lower than p99. A few slow failing requests barely change the average but dominate the tail, which is why percentiles are used.

Operating a log cluster

  • Retention: decide how many days you must keep, and let lifecycle policies delete or move older data. Deleting a whole backing index is far cheaper than deleting documents one by one (segments lesson).
  • Ingest rate: use bulk requests, a larger refresh_interval (for example 30 s), and size shards for daily volume (query performance).
  • Cardinality: never use unbounded values (such as full URLs with IDs, or session IDs) as grouping fields; a terms aggregation on millions of distinct values is slow and memory hungry. Normalize paths first (/product/123 becomes /product/:id).
  • Sensitive data: logs often contain emails, tokens or IP addresses. Scrub them in the ingest pipeline and restrict access with roles (security and operations).
  • Alerting: alert on the error rate and latency percentiles, not on single errors.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why parse log lines into fields at ingest time?

    Parsing once makes every later filter, histogram and top-N cheap, and lets you use proper types for numbers and dates.
  2. 2. What does a date_histogram aggregation do?

    Each document falls into a time bucket, and you can nest other aggregations inside each bucket.
  3. 3. Why do dashboards show p95 or p99 latency instead of only the average?

    A few very slow requests barely move the mean, but they define the tail that users notice.
  4. 4. Which field choice is dangerous for a terms aggregation on a large log index?

    Unique values create enormous numbers of buckets. Normalize them or group by a low-cardinality field.

Summary

  • Parse log lines at ingest into typed fields (dissect/grok, date, convert), and keep failures visible.
  • Use data streams, index templates and lifecycle policies for append-only time-series data.
  • Search with filters (level, service, time range); build dashboards with date_histogram, terms and percentiles.
  • Read a dashboard as when, where and how bad: histogram, top-N, then rate and percentiles.
  • Watch cardinality, retention and sensitive data.

Next: Project: Build a Mini Search Engine.