Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 27 ยท PromQL

Quantiles, Histograms and SLI Queries

Turn histogram buckets into latency percentiles with histogram_quantile, write error-rate and availability queries, and learn how accurate the estimates really are.

Averages hide pain. If 99 requests take 50 ms and one takes 10 seconds, the average looks fine while one user in a hundred is miserable. That is why latency is described with quantiles: the 95th percentile is the time that 95 percent of requests beat. Prometheus can compute quantiles from histograms, and this lesson shows the query, the arithmetic behind it, and the limits of the answer.

The second half of the lesson covers the other numbers that matter to users: the error rate, the availability ratio and the Apdex score. Together they are your SLIs (service level indicators), the measurements that an SLO (a target such as "99.9 percent of requests succeed") is checked against. If you need a refresher on how histogram series are laid out, read Counters, Gauges, Histograms and Summaries first, and for the counter math used below see rate, increase and Counter Math.

Prometheus, Grafana and the queries in this lesson cannot run on this site, so the PromQL appears in read-only blocks. The runnable examples are Python stand-ins that implement the same arithmetic on small data sets, so you can see what histogram_quantile does inside. They are not Prometheus itself.

One histogram, three kinds of series

_bucket Cumulative count per upper bound (le).
_sum Total of all observed values.
_count How many observations happened.

Buckets are counters, so you apply rate() before anything else.

1 / 2

What a histogram exposes

A histogram metric such as http_request_duration_seconds is a family of series. For each upper bound there is a counter named ..._bucket with a label le ("less than or equal"). The counts are cumulative: the bucket le="0.25" counts every request that took 0.25 seconds or less, including those already counted in le="0.1". The last bucket is le="+Inf" and equals the total count.

TEXT
http_request_duration_seconds_bucket{le="0.05"} 24
http_request_duration_seconds_bucket{le="0.1"} 60
http_request_duration_seconds_bucket{le="0.25"} 90
http_request_duration_seconds_bucket{le="0.5"} 98
http_request_duration_seconds_bucket{le="1"} 100
http_request_duration_seconds_bucket{le="+Inf"} 100
http_request_duration_seconds_sum 21.4
http_request_duration_seconds_count 100

Notice that nothing here stores individual requests. Prometheus only knows how many fell into each bucket, and that is both the strength (cheap, mergeable across servers) and the weakness (the answer is an estimate).

The histogram_quantile query

Because buckets are counters that only ever grow, a quantile over all history would barely move. You want the quantile over recent traffic, so you take the per-second rate of each bucket over a window first:

PROMQL
histogram_quantile(
  0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

Read it from the inside out. rate(...[5m]) turns each bucket counter into "requests per second that landed here, over the last five minutes". sum by (le) adds the series of all instances and all other labels together, keeping only le. That step is what makes histograms aggregatable: you can compute a fleet-wide p95 from per-pod buckets, which you can never do correctly by averaging per-pod percentiles. Finally histogram_quantile(0.95, ...) finds the bucket where the 95th percentile falls.

If you want the p95 per route, keep another label: sum by (le, route) (...). The le label must always survive the aggregation or the function has nothing to work with.

How the estimate is made

The function takes the total, multiplies by the quantile to get a target rank (for q = 0.95 and 100 requests, rank 95), and finds the first bucket whose cumulative count reaches that rank. Inside that bucket it assumes the observations are spread evenly and interpolates linearly between the bucket's lower and upper bound. The first bucket's lower bound is taken as 0. If the rank falls in the +Inf bucket, there is no upper bound to interpolate toward, so the function returns the highest finite bound.

Try the same arithmetic yourself.

Python Interpolating a quantile from buckets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The median lands in the 0.05 - 0.1 bucket, p90 is exactly on the edge of the 0.25 bucket, and p95 sits in the 0.25 - 0.5 bucket where the interpolation guesses it. That guess is the subject of the next section.

Choosing buckets and knowing the limits

The estimate can never be more precise than the buckets. If the true p95 lies somewhere between 250 ms and 500 ms, Prometheus reports a value in that range, and the real number could be anywhere inside it. Two practical rules follow:

  • Put bucket boundaries around your SLO. If the goal is "95 percent of requests under 300 ms", add a bucket at exactly 0.3. Then the SLI is not an estimate at all: it is the ratio of the le="0.3" bucket to the total.
  • Cover the range you expect. Everything beyond your largest finite bucket collapses into +Inf and the quantile gets stuck at that largest bound. A service with 30-second timeouts needs buckets that reach 30 seconds.

Very wide buckets give flat, misleading graphs, and very many buckets multiply your series count: each bucket is one series per label combination, and a label with many values multiplies it again (see Metrics, Labels and Time Series for cardinality). Ten to fifteen well-placed buckets is a common starting point. Native histograms, a newer Prometheus feature, use automatically sized exponential buckets and reduce this trade-off, but they need support on both the application and the server side, so check your versions before relying on them.

A last warning: you cannot average quantiles. The p95 of ten pods is not the mean of ten p95 values. Always sum the buckets first and compute the quantile once, as in the query above. Summary metrics, by contrast, precompute quantiles in the application and cannot be merged, which is why histograms are usually preferred for services with several instances.

Error rates and availability ratios

Not every SLI is latency. A request-based availability SLI is "the share of requests that succeeded", which is a ratio of two rates. Given a counter http_requests_total with a status label:

PROMQL
# Fraction of requests that failed over the last 5 minutes
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

Both sides use the same window, so the ratio is stable even when traffic changes. The availability SLI is simply 1 - that expression. A few habits make these queries safer:

  • Sum the numerator and denominator separately before dividing, so the result is one ratio for the service and not a different ratio per series.
  • Use the same label filters on both sides, except for the part that defines "bad".
  • When traffic can drop to zero, the division returns nothing (an empty result) rather than 0. That is usually correct: no requests means no information. Alert rules should treat "no data" deliberately (more in Alerting Rules and Alertmanager).
  • Decide whether 4xx responses count as errors. Usually they are the client's mistake and do not count against availability, but a sudden surge of 401 or 404 can still point to a broken deploy.

The same bucket trick gives a latency SLI without any estimation, because the le bucket at your threshold is a plain counter:

PROMQL
# Share of requests faster than 300 ms
sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))

Apdex

Apdex condenses latency into one score between 0 and 1. You choose a target time T. Requests faster than T are satisfied, requests between T and 4T are tolerating and count half, and slower requests are frustrated and count nothing. With buckets at T and 4T the score needs no interpolation:

PROMQL
(
  sum(rate(http_request_duration_seconds_bucket{le="0.25"}[5m]))
  +
  sum(rate(http_request_duration_seconds_bucket{le="1"}[5m]))
) / 2
/
sum(rate(http_request_duration_seconds_count[5m]))

The le="1" bucket already contains the satisfied requests, so adding the two buckets and halving counts satisfied once and tolerating half. Apdex is easy to put on a dashboard, but a single number hides why it dropped, so treat it as a summary beside the percentile and error graphs.

Python Error rate, availability and Apdex from counter snapshots

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The stand-in does by hand what rate() does in Prometheus: the increase divided by the window length. The real function also handles counter resets and extrapolates to the window edges, which rate, increase and Counter Math explains.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is the query sum by (le) (rate(..._bucket[5m])) used before histogram_quantile?

    Buckets are ever-growing counters, so rate() gives recent behavior, and summing by le combines series from all instances so one fleet-wide quantile can be computed.
  2. 2. A service's real p95 is 320 ms, but the nearest buckets are le="0.25" and le="0.5". What does histogram_quantile(0.95, ...) return?

    The function assumes observations are spread evenly inside the bucket and interpolates. The result is only an estimate within the bucket's range.
  3. 3. You want an exact SLI for 'requests faster than 300 ms'. What is the best preparation?

    A bucket boundary exactly at the SLO threshold makes the SLI a plain ratio of two counters, with no interpolation error.
  4. 4. Why should you not average the p95 values of several instances?

    A mean of percentiles is not the percentile of the combined traffic, especially when instances serve different amounts of load.
  5. 5. What happens to histogram_quantile when the requested rank falls in the le="+Inf" bucket?

    There is no upper bound to interpolate toward, so the largest finite bound is returned. Add buckets that cover your timeouts to avoid this.

Summary

  • Histogram buckets are cumulative counters labelled le; apply rate() first, then sum by (le), then histogram_quantile.
  • The quantile is an interpolated estimate, so its accuracy depends on bucket placement; put boundaries at your SLO thresholds and cover your timeouts.
  • Never average quantiles across instances; merge buckets instead.
  • Error rate and availability are ratios of summed rates over the same window; a bucket at the threshold gives an exact latency SLI.
  • Apdex combines the satisfied and tolerating buckets into one score, best shown next to percentile and error graphs.

Next: Recording Rules and Query Performance