Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 37 ยท Queries and Transformations

Prometheus Queries in Grafana: rate, Legends and $__rate_interval

Write PromQL in Grafana: builder versus code mode, rate and increase on counters, $__rate_interval, legend formats, instant versus range queries and table output.

Prometheus is the data source most Grafana users meet first, and its query language, PromQL, has a few habits that matter inside Grafana. This lesson covers the query editor for Prometheus, how to graph counters correctly with rate(), why $__rate_interval exists, and how to shape the legend and the output format.

Prometheus and Grafana are not available on this site, so the examples are shown as query text, and the runnable blocks are Python stand-ins that perform the same arithmetic on sample numbers. Treat their output as an illustration of the math, not as Grafana output.

Builder and code mode

The Prometheus query editor has a Builder with a metric dropdown, label filters, and an Operations menu (Rate, Sum, Histogram quantile, and so on), plus a Code mode with a text box that offers autocomplete and a Metrics browser. The builder shows a live preview of the generated query. Beginners use the builder; people who know PromQL work in code mode. A query typed in code mode is parsed back into the builder when possible.

Below the query box you find options for the row:

  • Legend: Auto, Verbose, or Custom, where you write a template such as {{instance}} - {{method}}. Each {{label}} is replaced by that series' label value.
  • Min step: the lowest allowed query resolution (the step parameter).
  • Format: Time series, Table or Heatmap.
  • Type: Range, Instant or Both.
  • Exemplars: toggles points that link a metric spike to a trace.

Counters and rate()

Most application metrics are counters: numbers that only grow (for example http_requests_total). Plotting a raw counter gives a line that climbs forever and resets to zero when the process restarts. What you actually want is the per-second rate of increase:

PROMQL
rate(http_requests_total{job="api"}[5m])

rate() looks at the samples inside the last five minutes for each series, handles counter resets, and returns requests per second. A few companions to know:

  • increase(metric[5m]) is the total increase over the window (about rate * 300).
  • irate(metric[5m]) uses only the last two samples, so it reacts fast and looks spiky; it suits zoomed-in graphs.
  • sum by (status) (rate(...)) aggregates away every label except status.
Python Stand-in: rate() with a counter reset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real Prometheus also extrapolates to the window edges, so its numbers differ slightly; the point here is how resets are handled.

$__rate_interval

How wide should the [5m] range be? If it is shorter than about two scrape intervals, rate() has fewer than two samples and returns nothing. If it is a fixed 5m but you zoom into 10 seconds, the graph is smoothed beyond recognition. Grafana's answer is the built-in variable $__rate_interval:

PROMQL
rate(http_requests_total{job="api"}[$__rate_interval])

It is calculated as the larger of $__interval + scrape interval and four times the scrape interval. For this to work, set the Scrape interval in the Prometheus data source settings to the real value (15s by default in Prometheus). Use $__rate_interval for rate, increase and irate; use $__interval for aggregations over time such as avg_over_time.

Python Stand-in: compute $__rate_interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Legends and series names

Without a legend format, a series is named by all its labels, for example {instance="10.0.0.5:9100", job="node", mode="idle"}. That is unreadable. Use custom legends:

  • {{instance}} for one series per host.
  • {{method}} {{status}} for request breakdowns.
  • Leave the legend empty and use Standard options > Display name to rename series in the panel instead.

Another trick for legends: hide noisy labels by aggregating them away with sum by (...) or without (...), so the series count stays small.

Instant versus range queries

A range query asks Prometheus for a value at every step across the panel's time range, which is what time series graphs need. An instant query returns just the value at the end of the range. Use instant for Stat, Gauge, Bar gauge and Table panels, because you want one number per series and the instant query is cheaper. Choosing Both gives you a graph and a table from the same query.

Table and heatmap formats

Set Format to Table when you want one row per series with labels as columns, which suits a "top 10 pods by memory" table. The result has columns such as instance, Time and Value #A; organize them with the transformations in the next lessons. Set Format to Heatmap for histogram buckets, which Grafana turns into a bucketed color grid when the metric has le labels:

PROMQL
sum by (le) (rate(http_request_duration_seconds_bucket[$__rate_interval]))

For a latency percentile you can instead use:

PROMQL
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[$__rate_interval])))

Useful patterns

  • Error ratio: sum(rate(http_requests_total{status=~"5.."}[$__rate_interval])) / sum(rate(http_requests_total[$__rate_interval])).
  • Memory used percent: 100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes).
  • Dashboard variable in a label filter: {instance=~"$instance"}, so multi-select variables work (see dashboard variables).
  • Top N: topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total[$__rate_interval]))).

A common mistake is to graph rate() of a gauge (such as memory usage). rate() is only meaningful for counters; gauges are plotted directly or with avg_over_time.

flowchart LR
    C["Counter<br/>http_requests_total"] --> R["rate(...[$__rate_interval])"]
    R --> S["sum by (status)"]
    S --> L["Legend: {{status}}"]
    L --> G["Time series panel"]

A typical counter panel: rate first, then aggregate, then name the series.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do you wrap a counter such as http_requests_total in rate()?

    A raw counter only grows and drops to zero on restarts; rate() gives the per-second increase and handles resets.
  2. 2. What is the main reason to use $__rate_interval instead of a fixed [5m]?

    It is at least four scrape intervals wide and grows with the panel's step, so rate() works at any zoom.
  3. 3. Which query type suits a Stat panel that shows one number per series?

    An instant query returns only the latest value, which is cheaper and exactly what a single-value panel needs.
  4. 4. What does the legend format {{instance}} do?

    Each {{label}} placeholder is replaced by that series' label value.

Summary

  • Prometheus rows offer builder and code modes, legend templates, min step, format and range or instant type.
  • Use rate() or increase() on counters, never on gauges; use sum by to keep series counts small.
  • $__rate_interval keeps rate() working at every zoom level when the data source scrape interval is correct.
  • Instant queries and table format suit single-value and table panels; heatmap format suits histogram buckets.
  • Reuse dashboard variables in label matchers with =~ so multi-select works.

Next: SQL and Loki Queries in Grafana