Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 27 ยท PromQL

rate, increase and Counter Math

Understand how rate, irate and increase work on counters: resets, window size, extrapolation and the common mistakes that give wrong graphs and alerts.

A counter only goes up. Its raw value, say http_requests_total 8,413,502, tells you almost nothing: it depends on when the process started and says nothing about now. What you want is the speed at which it grows, requests per second over the last five minutes. The function that turns a counter into a speed is rate(), and it is probably the most used function in PromQL.

It looks simple, but a lot happens inside it: it survives restarts (counter resets), it never sees data between samples, and it makes educated guesses at the window edges. Most wrong dashboards and flapping alerts come from not knowing these details. This lesson explains rate, irate and increase, what window to choose, and the traps to avoid. The queries are read-only examples; the runnable parts are Python stand-ins that implement the same algorithm so you can check the arithmetic.

The three functions

Three ways to read a counter

rate Average per-second growth over the window.
irate Per-second growth between the last two samples.
increase Total growth over the window.
1 / 2

All three take a range vector of a counter and return an instant vector:

TEXT
rate(http_requests_total{job="api"}[5m])       # requests per second, averaged over 5 minutes
increase(http_requests_total{job="api"}[1h])   # requests in the last hour
irate(http_requests_total{job="api"}[5m])      # per second, from the last two samples only

rate is the average speed over the window. increase is the same quantity without dividing by the window length, so it answers "how many in the last hour?". irate ignores everything in the window except the two newest samples, which makes it jumpy: it is the instantaneous speed, good for zooming in on short spikes in a graph, but poor for alerts, because one odd pair of samples decides the value. The window only has to be long enough to contain at least two samples; the rest is thrown away.

Counter resets

When a process restarts, its counter starts again from zero. A naive "last value minus first value" would come out negative. rate and increase detect this: whenever a sample is lower than the previous one, they assume the counter was reset to zero and that the whole new value counts as growth since the reset.

TEXT
samples: ... 180, 210, 20, 50, 80 ...
                       ^ reset: counted as +20, not -190

This works only if the counter really restarts at zero and every increase between samples is smaller than the value before the reset. If a restart is quick and the counter climbs past its old value between two scrapes, the reset is invisible and the increase is underestimated, but in practice that is rare. It is one more reason to scrape every 15 to 60 seconds and not much less often.

Never take rate() of a gauge. A gauge can go down legitimately, and rate would read each decrease as a reset and invent traffic. Use deriv() or delta() for gauges instead.

Extrapolation

Samples rarely line up exactly with the window edges. If you scrape every 15 seconds and ask for the last minute, the first sample in the window might be 7 seconds after the window start. If rate only used the time between the first and last sample, it would slightly underestimate. So Prometheus extrapolates: it extends the observed increase to cover the full window, but only up to half a scrape gap past the edges (or fully to an edge if the first or last sample sits within about 1.1 times the average gap of it), and never beyond the point where the counter would have been below zero.

Two consequences matter in practice:

  • increase() of an integer counter usually returns a fractional number (such as 25.56), because it is extrapolated. If you need exact counts, count in a different way (a recorded total, or the difference of two instant values).
  • The division always uses the requested window, not the span of the data. If a window is longer than the history that exists, rate underestimates. After a fresh start, a [5m] rate over one minute of data shows roughly a fifth of the true value.

Choosing the window

A window is a number of seconds, but rate needs samples. Use at least four times your scrape interval: with a 15 s scrape, [1m] is a sensible minimum, and [5m] is the common default. Too short a window and one missed scrape leaves a single sample, so rate returns nothing and graphs get holes. Too long a window and the result is smooth but late: a traffic spike takes minutes to show up and just as long to fade.

Window Behavior
Equal to the scrape interval Often returns an empty result (one sample)
About 4x the interval Reacts quickly, tolerates one missed scrape
5 minutes Typical for dashboards and alerts
1 hour and more Good for slow trends and capacity planning; too slow for alerts

In Grafana, use the built-in variable $__rate_interval in the window, as in rate(x[$__rate_interval]). It picks a window that adapts to the dashboard zoom and always covers at least four scrape intervals, so zoomed-in graphs do not suddenly turn empty.

Rate math in practice

Aggregate after rate, never before. Summing raw counters first would hide resets in the individual series and give wrong results:

TEXT
sum(rate(http_requests_total[5m]))              # correct: total request rate
sum by (job) (rate(http_requests_total[5m]))    # per job

rate(sum(http_requests_total)[5m:])             # wrong in spirit: sum of counters can hide a reset

A few widely used patterns:

TEXT
# CPU usage percent per machine, from the idle counter
100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100

# Network traffic in bits per second
rate(node_network_receive_bytes_total{device="eth0"}[5m]) * 8

# Share of requests that failed
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# Average latency from a histogram's sum and count
rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])

A related function is resets(x[1h]), which counts how many times a counter dropped, useful for spotting crash loops. The function increase is perfect for alert conditions like "more than 10 errors in the last 10 minutes", written increase(errors_total[10m]) > 10.

The algorithm, runnable

This Python stand-in implements the algorithm described above (the real one lives in Prometheus's Go engine, which has more edge cases). It computes rate, increase and irate for a counter scraped every 15 seconds, with and without a restart. The window is left-open, as in Prometheus 3.0 and later.

Python rate, increase and irate with a counter reset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The steady counter grows by 30 every 15 seconds, so the true speed is exactly 2 per second, and rate returns 2.0. For the restarted counter, the drop from 210 to 20 is read as a reset, and 20 requests are counted for the interval in which it happened. The naive difference would be negative.

Window size and common mistakes

The second stand-in reproduces the traps one by one: windows that are too short or longer than the data, the jumpiness of irate, and a counter that only exists after its first event. It uses the same functions as before.

Python Window traps: too short, too long, bursts and new series

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output with the lessons in mind:

  • A [15s] window holds one sample, so there is no result at all. [30s] already works, but real data needs margin for missed scrapes.
  • A [5m] window over only 60 seconds of data returns 0.4 instead of 2, because the division uses the full five minutes.
  • The burst shows the difference between rate and irate. The 15-second spike produces irate of 40 per second for exactly one scrape and then 0. rate[1m] stays at 13.33 per second for as long as the burst is inside the window, then drops to zero. Alerts want the second behavior.
  • A series that is created by its first event misses that first increment: the increase is 2.33 instead of 3. This is why client libraries should initialise known label values at zero.
  • An integer counter yields a fractional increase (25.56, where the raw difference is 23), because of extrapolation.

Other frequent mistakes: using rate() on a gauge, taking sum() before rate(), using irate in alert rules, writing windows shorter than two scrapes, and forgetting that rate() of a histogram bucket needs the le label preserved when you later call histogram_quantile (see Quantiles, Histograms and SLI Queries).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does rate(http_requests_total[5m]) return?

    rate is the average per-second growth across the window. increase is the total, and irate uses only the last two samples.
  2. 2. A process restarts and its counter drops from 1,000 to 40. How do rate and increase treat this?

    A sample lower than the previous one is interpreted as a counter reset, so the values after the restart are added to the growth.
  3. 3. You scrape every 30 seconds. Which window is the safest minimum for rate()?

    Use roughly four scrape intervals so that the window holds several samples even if a scrape is missed. A window of one interval often contains a single sample, which gives no result.
  4. 4. Which expression computes the total request rate across all instances correctly?

    Compute the rate per series first, so resets in individual series are handled, then aggregate. Summing counters first can hide resets.
  5. 5. Why is irate a poor choice for alert rules?

    irate is the instantaneous rate and is very spiky. Alerts need the stability of rate over a window.

Summary

  • rate is the per-second average over a window, increase the total over it, irate the speed between the last two samples.
  • Counter resets are detected when a value drops; the new value counts as growth. Never use rate on gauges.
  • Prometheus extrapolates to the window edges, so increase of an integer counter can be fractional, and a window longer than the available data underestimates.
  • Choose windows of at least four scrape intervals; use $__rate_interval in Grafana; use rate for alerts and irate only to inspect spikes.
  • Always aggregate after rate, initialise known counters at zero, and divide rate of a _sum by the _count rate for averages.

Next: combine and join series with Aggregation, Joins and Vector Matching.