Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 27 ยท PromQL

Aggregation, Joins and Vector Matching

Combine time series in PromQL: sum, avg, topk and count with by and without, binary operators, on and ignoring, group_left joins, and absent().

A single service can produce thousands of time series: one per instance, per route, per status code. You rarely want to look at them one at a time. You want "requests per second per job", "the five busiest instances", or "the error ratio of each service". PromQL does this with two families of tools: aggregation operators that collapse many series into fewer, and binary operators that combine two sets of series, matching them up by their labels.

Both are easy to use and easy to get subtly wrong. The runnable parts of this lesson are Python stand-ins that implement the same grouping and matching rules on small data, so you can see exactly which series are paired, dropped or rejected. The PromQL examples themselves are read-only, since no Prometheus server runs on this site.

Aggregation operators

Collapse many series into few

sum Total across series.
avg Mean across series.
min and max The extremes.
count How many series there are.
1 / 2

The core operators are sum, avg, min, max and count, plus topk, bottomk, quantile, stddev, stdvar and count_values. Each takes an instant vector and returns another. The grouping clause says which labels identify a group:

TEXT
sum(rate(http_requests_total[5m]))                        # one number: all traffic
sum by (job) (rate(http_requests_total[5m]))              # one value per job
sum without (instance, status) (rate(http_requests_total[5m]))   # same, by exclusion

by (job) keeps only the job label; every other label is thrown away, and series that share the same job are added together. without (instance, status) does the opposite: it removes the listed labels and keeps everything else. Use by when you know exactly which dimensions you want; use without when new labels might appear later and you want them to keep passing through. In both cases the metric name (__name__) is dropped, since the result is no longer the original metric.

Three more things to remember:

  • count counts series, not values. count by (job) (up) is the number of targets per job, and count(up == 0) is the number of down targets.
  • avg of averages is wrong. Averaging per-instance averages gives each instance equal weight regardless of traffic. For a true mean, divide a sum by a count (for latency, rate(..._sum) / rate(..._count)).
  • topk(3, expr) returns the three series with the largest values, with their original labels. In a graph the "top" set can change at every step, so panels may show series that appear and vanish; use it on instant queries and tables, or combine it with a long window.
TEXT
topk(5, sum by (instance) (rate(http_requests_total[5m])))   # five busiest instances
count by (version) (app_build_info)                          # how many instances run each version

Binary operators and vector matching

PromQL supports arithmetic (+ - * / % ^), comparison (== != > < >= <=) and logical/set operators (and or unless) between expressions. Between a vector and a scalar, the operator applies to every sample. Between two vectors, Prometheus must decide which series on the left belongs with which series on the right. By default it pairs series that have exactly the same labels (ignoring the metric name). That is why this works:

TEXT
sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
  /
sum by (job) (rate(http_requests_total[5m]))

Both sides have just a job label, so {job="api"} pairs with {job="api"}. The result is the error ratio per job. Two important rules follow from the default matching:

  1. A series without a partner on the other side is silently dropped from the result. If a job has had no errors, there is no {job="web"} series on the left, so web disappears from the ratio instead of showing 0. Fix this by adding a fallback with or, for example (errors or total * 0) / total.
  2. A comparison filters by default: up == 0 keeps only the series where the condition is true, and returns their values (0). Add bool (up == bool 0) to get a 0 or 1 for every series instead.

on and ignoring

When the two sides have different label sets, tell Prometheus which labels to compare:

TEXT
errors_total / on (job) requests_total              # match using only the job label
errors_total / ignoring (status) requests_total     # match using every label except status

on(...) lists the labels that must be equal; ignoring(...) lists labels that are left out of the comparison. After the match, the result keeps the labels of the left-hand series for one-to-one matches.

group_left and group_right

Often one side has more series per match than the other. The classic case is attaching information from an "info" metric, a gauge that is always 1 and carries labels:

TEXT
app_build_info{instance="a:9102", version="1.4.2"} 1
app_build_info{instance="b:9102", version="1.5.0"} 1

To label per-instance traffic with the version, you multiply by the info metric and ask for a many-to-one match:

TEXT
sum by (instance) (rate(http_requests_total[5m]))
  * on (instance) group_left (version)
app_build_info

on (instance) pairs by instance. group_left says the left side is the "many" side and may have several series per partner (here the left could even have multiple series per instance), while (version) names the labels to copy over from the "one" side. group_right is the mirror image. Without a grouping modifier, a many-to-one match is an error ("many-to-one matching must be explicit"), and a one-to-many or duplicate on the "one" side is rejected too ("found duplicate series for the match group"). Multiplying by an info metric whose value is 1 changes nothing numerically, which is why it is the standard way to join labels.

Set operators

and, or and unless work on the label sets, not on values. a and b keeps the series of a that also exist in b, a or b returns all of a plus those of b without a counterpart in a, and a unless b keeps those of a that have no match in b. A typical use is silencing one alert while another is true, or finding what is missing:

TEXT
up == 0 unless on (instance) maintenance_mode == 1

absent()

A query that matches nothing returns an empty result, which an alert rule treats as "all fine". That is dangerous when the thing you wanted to watch has vanished. absent() turns emptiness into a signal:

TEXT
absent(up{job="payments"})                       # returns {job="payments"} 1 when no such series exists

absent(x) returns a single series with the value 1 (and any equality matchers of the selector as labels) when the vector x is empty, and nothing otherwise. For a range selector there is absent_over_time(x[10m]), which is better for series that appear only occasionally. Use absent to detect disappearing targets and silent pipelines, and use up == 0 to detect a target that exists but fails to respond; they catch different failures.

Aggregation in action

This stand-in implements the grouping rules in Python on a small instant vector of request rates: six series with job, instance and status labels. It is a model of the semantics, not Prometheus's engine.

Python sum, avg, max, count and topk with by and without

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The by (job) and without (instance, status) results are identical here because job, instance and status are the only labels, but only without would keep a new label such as region if one appeared. count by (job) returns the number of series, not the sum of values, and topk keeps every original label while sum by strips them.

Matching, joins and absent

The second stand-in implements one-to-one matching with on and ignoring, many-to-one with group_left, the "dropped without a partner" rule and its or fix, and absent. The error messages are the stand-in's own wording.

Python Error ratio, group_left and absent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Four things to notice. The first ratio contains only api: web had no 5xx series, so it had no partner. Adding the or fallback brings web back with 0. The group_left (version) result carries the version label copied from the info metric while keeping the values multiplied by 1. Without group_left the match is refused, because several left-hand series map to one right-hand series. And ignoring (status) pairs each 500 series with the 200 series of the same job and instance, which is useful for ratios between two statuses.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does sum by (job) (rate(http_requests_total[5m])) return?

    by (job) keeps only the job label and adds up all series that share it. Without by or without, you would get one single result.
  2. 2. You compute errors / total per job, and a job that never had errors is missing from the result. Why?

    With default one-to-one matching, a series without a partner is dropped. Add a fallback such as (errors or total * 0) / total to get an explicit 0.
  3. 3. Which modifier copies the version label from app_build_info onto per-instance series?

    on (instance) pairs the series, group_left says the left side has many series per partner, and its label list names the labels to copy from the right side.
  4. 4. What does count by (job) (up) return?

    count counts series, not their values. It is the number of targets per job, whether they are up or down.
  5. 5. When is absent(up{job="payments"}) useful?

    An empty result looks healthy to an alert rule. absent() returns 1 when the selector matches nothing, so you can alert on disappearance. up == 0 only catches targets that exist but fail.

Summary

  • sum, avg, min, max and count collapse series; by keeps listed labels, without drops them, and with neither you get one result. topk keeps original labels.
  • Vector operators match series with identical labels by default; unmatched series are dropped silently, so add an or fallback when zero must show up.
  • on and ignoring choose which labels to compare; group_left and group_right allow many-to-one matches and copy labels from the "one" side, the standard way to join info metrics.
  • Comparisons filter unless you add bool; and, or and unless work on label sets.
  • absent() and absent_over_time() turn an empty result into an alertable signal.

Next: measure latency distributions with Quantiles, Histograms and SLI Queries.