Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 27 ยท Recording Rules and Alerting

Recording Rules and Query Performance

Precompute expensive PromQL with recording rules: rule groups, the level:metric:operations naming scheme, when to use them and how to spot slow queries.

A PromQL query is evaluated from scratch every time it runs. For a small server that is fine. But a dashboard with twenty panels, refreshed every ten seconds by five people, asks Prometheus to read the same millions of samples over and over, and the same expression may also run inside several alert rules. A recording rule solves this: Prometheus evaluates the expression on a schedule and stores the result as a new, cheap time series that queries can read instead.

This lesson explains how rule files are structured, how to name the results so that people can read them, when precomputing helps (and when it only adds clutter), and how cardinality decides whether a query is slow in the first place. It builds on the aggregation ideas from Aggregation, Joins and Vector Matching.

Prometheus cannot run on this site, so rule files are shown in read-only blocks. The runnable examples are JavaScript and Python stand-ins that count how many samples a query has to touch, which is the main driver of query cost. They are a simplified model, not Prometheus's real engine.

What a recording rule is

A rule file is YAML. Rules live inside groups; Prometheus evaluates the rules of one group in order, at a fixed interval, and writes each result back into its own storage as a new series.

YAML
groups:
  - name: http_requests
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_request_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          job:http_requests:rate5m

You load the file from prometheus.yml and reload the server (a SIGHUP, or the /-/reload endpoint when --web.enable-lifecycle is on):

YAML
rule_files:
  - "rules/*.yml"

A few details are worth knowing:

  • Order inside a group matters. The second rule above reads the series created by the first one, which is only guaranteed to be fresh because they share a group and run sequentially. Rules in different groups run independently and concurrently.
  • interval is optional and defaults to the global evaluation_interval. Choose an interval at least as long as the scrape interval; evaluating faster than you scrape only repeats values.
  • Labels listed in by (...) survive; everything else disappears. The new series is a normal series: you can graph it, alert on it, and even use it in other rules.
  • A rule can also add or override static labels with a labels: block, for example team: payments.

Naming: level, metric, operations

Without a convention, a rules file turns into a pile of names like my_cool_rate that nobody can trace back to a source. The widely used convention has three parts separated by colons:

TEXT
level : metric : operations
  • level is the labels the result is aggregated by (job, instance, cluster, job_route).
  • metric is the original metric name, with _total dropped for counters that went through rate().
  • operations lists what was applied, newest first (rate5m, sum, ratio_rate5m).

So job:http_requests:rate5m reads: "per job, the request metric, as a 5-minute rate". Two more habits keep rules honest: always keep the original labels you aggregate by in the level part, and when you divide two series use a name that says so (ratio). A reader should be able to guess the expression from the name.

When to precompute

Recording rules are a tool, not a default. They help in four situations:

  1. Dashboards. Panels that run an expensive aggregation on every refresh, over long time ranges.
  2. Alert rules. An alert expression evaluates every minute or faster; a rule that reads a precomputed series is both faster and shows up as a series you can graph while debugging.
  3. Reuse. Several panels or alerts share the same sub-expression.
  4. Long-range queries. rate(x[5m]) over thirty days has to recompute 30 days of windows, while a recorded series is already computed.

They do not help when:

  • the query is already cheap (a single series selector needs no rule);
  • you only run it once or twice by hand while investigating;
  • the result would be as many series as the input. A rule that keeps all labels stores a second copy of the same data, which costs disk and memory and speeds up nothing.

A useful trick is a two-stage design: record the rate with the labels you need (job:...:rate5m), then build ratios and quantiles on top of that recorded series. The raw high-cardinality series is then read once per evaluation, not once per panel.

Another limit to remember: recording rules only exist from the moment you create them. They do not backfill history, so a new rule has no data before today unless you use the backfill tooling (promtool tsdb create-blocks-from rules).

Why queries get slow

A query's cost grows roughly with the number of samples it reads: the series matched, times the number of samples per series in the range. Three things multiply that number:

  • Cardinality. sum(rate(http_requests_total[5m])) over 5,000 series reads five thousand series, no matter that the answer is one number.
  • Range. A longer graph range, or a larger window such as [1h], reads more samples per series.
  • Resolution. A dashboard asks for one point per step; a small step over a long range means many evaluations.

Signs of trouble: panels that time out ("query processing would load too many samples"), or prometheus_rule_group_last_duration_seconds approaching the group's interval. The last one matters: if a group takes longer than its interval to evaluate, Prometheus skips evaluations and your recorded series and alerts get gaps. Prometheus exposes prometheus_rule_group_iterations_missed_total for exactly this case.

Typical fixes, in order of effort: narrow selectors with label matchers, shorten ranges, drop labels you never query (see Service Discovery and Relabeling), add a recording rule, and finally fix the cardinality at its source (see Metrics, Labels and Time Series).

A cost model you can run

The stand-in below does not talk to Prometheus. It models a service with a given number of instances, routes and status codes, and counts how many raw samples a dashboard panel must read per refresh, with and without a recording rule.

JS Raw query cost versus a precomputed series

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last loop shows the real trade-off. The rule has a fixed daily cost because it evaluates all day whether anyone looks or not. If the panel is opened ten times a day, precomputing is a net loss; for a wall-mounted dashboard that refreshes every few seconds, it is a big win. Always weigh the rule's cost against how often the result is actually read.

Cardinality: the other half

The model above multiplies three label dimensions. Add one more dimension with many values (a user ID, a full URL, a container ID) and the series count explodes by that factor, and recording rules cannot fix it: the rule still has to read every raw series at least once per evaluation. Counting the combinations is a useful habit.

Python How labels multiply series

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line, a "rule" that keeps every label, stores exactly as many series as it reads. It is the pattern to avoid. A good rule reduces cardinality sharply (by (route): 25 series from 100,000).

Check what is expensive before writing rules. In the Prometheus web UI the Status โ†’ TSDB Status page lists the metric names with the most series and the labels with the most values, which tells you where the cost really comes from.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a recording rule store?

    Prometheus evaluates the expression at each interval and writes the result into its own database, so later queries can read the cheap precomputed series.
  2. 2. What does the name job:http_requests:rate5m tell you?

    The convention is level:metric:operations: the labels kept, the source metric and the operations applied.
  3. 3. Two rules in the same group, where the second reads the series made by the first. Why does the order inside the group matter?

    Within a group rules are evaluated sequentially. Across groups there is no ordering guarantee.
  4. 4. Which recording rule is most likely a waste?

    A rule that keeps all labels stores as many series as it reads, so it costs storage and evaluation time without reducing query work.
  5. 5. Which signal suggests a rule group is too slow for its interval?

    If a group takes longer than its interval, iterations are missed. Metrics such as prometheus_rule_group_iterations_missed_total show it.

Summary

  • A recording rule evaluates a PromQL expression on a schedule and stores the result as a new series; rules are grouped, and rules in one group run in order.
  • Name results level:metric:operations so the expression can be guessed from the name.
  • Use rules for expensive, frequently read or shared expressions; skip them for cheap queries, one-off investigations and rules that keep every label.
  • Query cost follows the number of samples read: cardinality times range times resolution. Fix cardinality at the source when you can.
  • A rule has a fixed evaluation cost, so compare it with how often the result is read, and watch for missed group iterations.

Next: Alerting Rules and Alertmanager