Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 27 ยท PromQL

PromQL Basics

Learn PromQL from the ground up: instant and range vectors, label matchers, offset, subqueries, and how staleness and the lookback window decide results.

Prometheus stores data, but you get value out of it by asking questions. PromQL (Prometheus Query Language) is the language for that. It is used everywhere: in the expression browser, in Grafana panels, in recording rules and in alert conditions. PromQL is not SQL. It is a functional language over time series, where every expression returns one of a few well-defined result types.

Learn those types first and the rest follows. This lesson covers the building blocks you need before any function: selecting series, filtering them with label matchers, looking back in time with range vectors and offset, running a query over a window with subqueries, and the rules (the lookback window and staleness) that decide what an instant query returns. Examples of real queries are read-only, since no Prometheus server runs here. The runnable parts are stand-ins in Python that implement the same selection rules on sample data.

The four result types

Every expression has a type

Scalar One plain number, like 0.5.
Instant vector One value per series at one moment.
Range vector A list of samples per series over a window.
String Rarely used.
1 / 2

An instant vector is the result of a plain selector such as up. It holds one sample per matching series, all at the same evaluation time. A range vector looks like http_requests_total[5m] and holds all samples from the last five minutes for each series. Range vectors are the input of functions such as rate(), which reduce them back to an instant vector. A scalar is a single number. Knowing which type an expression returns explains most error messages: "expected type instant vector in aggregation" means you passed a range vector where an instant one was needed.

Selecting series

The simplest query is a metric name:

TEXT
http_requests_total

It returns every series with that name, one per label combination. To narrow the result, add label matchers in braces:

TEXT
http_requests_total{job="api", status="500"}
Matcher Meaning
= Label equals the value
!= Label is not equal to the value
=~ Label matches the regular expression
!~ Label does not match the regular expression

Two details cause real bugs. First, regexes are fully anchored: status=~"5" matches only the exact value 5, never 500; write status=~"5..". Second, a series that does not have a label is treated as having an empty value. That means status!="200" also returns series that have no status label at all, and status="" selects exactly those series. A selector must also contain at least one matcher that cannot match the empty string, so {job=~".*"} is rejected.

The metric name is itself a label called __name__, which allows selecting by name pattern:

TEXT
{__name__=~"http_.*_total", job="api"}

Use it for exploration only, because it touches many series and is slow on a big server.

The expression browser and the evaluation time

An instant query is always evaluated at one moment. In the Prometheus UI that is "now" unless you pick a time. A graph is just the same expression evaluated repeatedly at regular steps across a time range. This is why you see two kinds of API calls: query (one moment) and query_range (many moments with a step).

For a given evaluation time t, the instant vector selector up does this for every matching series: look back from t for the newest sample, and use it if it is no older than the lookback window, five minutes by default. That sample may be older than t by almost the full window. A series whose last sample is older than that is simply absent from the result.

Staleness

If the lookback window were the only rule, a target that vanished would still show its last values for five minutes. Prometheus avoids that with staleness markers: when a target disappears from service discovery, when a scrape fails, or when a series is no longer exposed by a target, Prometheus writes a special marker sample. An instant query that finds the marker as its newest sample returns nothing for that series, so the series ends cleanly at once. When no marker exists (data that was backfilled or written remotely, for example), the five-minute lookback is what ends the series.

Staleness is also why up == 0 alerts work, and why a graph that "falls off a cliff" and a graph that has "no data" are different situations.

Range vectors

Add a duration in square brackets to get a range vector:

TEXT
http_requests_total{job="api"}[5m]

Durations use s, m, h, d, w and y, and can be combined (1h30m). The result holds, for each series, the samples whose timestamps fall within the last five minutes before the evaluation time. In Prometheus 3.0 and later the window excludes its own start, so it covers the interval from five minutes ago (exclusive) up to now (inclusive). Older versions included the start timestamp, which could produce one more sample in the window.

The window is measured in time, not in number of samples. If you scrape every 15 seconds, [1m] holds about four samples; if you scrape every minute, it holds one or two. This matters a great deal for rate(), which needs at least two samples. The next lesson returns to that.

Looking back in time with offset

offset shifts the evaluation time of a selector into the past:

TEXT
http_requests_total{job="api"} offset 1h
http_requests_total{job="api"}[5m] offset 1d

It is the quickest way to compare now with before. The following expression gives the ratio of requests per second now versus the same moment last week, a quick "is this traffic normal?" check:

TEXT
rate(http_requests_total{job="api"}[5m])
  / rate(http_requests_total{job="api"}[5m] offset 1w)

The offset modifier must come directly after the selector, before any function call wraps it; rate(x[5m] offset 1w) is valid, while rate(x[5m]) offset 1w is not.

Subqueries

A normal range vector must be a selector. A subquery lets you take a range of any expression, by evaluating it repeatedly. The syntax is <expression>[<range>:<step>]:

TEXT
max_over_time(rate(http_requests_total{job="api"}[5m])[1h:1m])

This evaluates the inner rate(...) once a minute across the last hour (the :1m is the step), then takes the maximum of those hourly values: "the busiest 5-minute rate during the last hour." Without a subquery you would need a recording rule to store the inner result first. If you leave out the step ([1h:]), Prometheus uses its global evaluation interval. Subqueries are powerful but expensive, since each one multiplies the work, so use them for ad-hoc analysis rather than in dashboards that refresh every few seconds.

A selector engine you can run

This is a stand-in written in Python, not Prometheus's engine. It stores a few series with one sample per minute and implements label matchers (including anchored regexes and missing labels) and the instant vector selection with a five-minute lookback.

Python Selectors, matchers and the lookback window

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the queries with the output. status=~"5.." finds only the 500 series, and status=~"5" finds nothing because the regex is anchored. The last two lines show the lookback window ending: the final sample is at t=600, so it is still returned 299 seconds later but not at 300 seconds.

Range vectors, offset, staleness and subqueries

The second stand-in adds range vectors with a left-open window (as in Prometheus 3.0 and later), offset, stale markers (stored as None) and a subquery that evaluates an instant query on a grid of steps and takes the maximum.

Python Range vectors, offset, stale markers and a subquery

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the [1m] range at t=600 holds exactly one sample, because the scrape interval is a minute and the window excludes its start. That is why windows must be at least a few scrape intervals wide. The subquery evaluated the inner expression five times, and max picked the largest.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the expression http_requests_total{status=~"5"} return for series with status="500"?

    PromQL anchors regex matchers at both ends. To match 500 write status="5.." or status="5.*".
  2. 2. Which expression is a range vector?

    A selector followed by a duration in square brackets is a range vector. rate() and sum() return instant vectors.
  3. 3. A target is removed from service discovery. Why does its series disappear from queries immediately instead of lingering for five minutes?

    Staleness markers tell queries that the series ended. Without them the default five-minute lookback window would keep returning the last value.
  4. 4. What does the offset modifier do?

    offset 1h evaluates the selector as it would have looked one hour earlier, which makes comparisons with the past easy.
  5. 5. When do you need a subquery?

    Range vectors normally come only from selectors. A subquery [range:step] repeatedly evaluates any expression so functions like max_over_time can use it. It costs more, not less.

Summary

  • PromQL expressions return a scalar, an instant vector, a range vector or a string; functions like rate() turn range vectors into instant vectors.
  • Select with name{label="value"}; matchers are =, !=, =~ and !~, regexes are anchored, and a missing label equals the empty string.
  • An instant query returns the newest sample within the five-minute lookback window; staleness markers end series immediately.
  • [5m] makes a range vector (a time window, not a number of samples), offset evaluates in the past, and [1h:1m] is a subquery.
  • Choose windows that hold several scrapes, and keep subqueries out of fast-refreshing dashboards.

Next: turn counters into rates with rate, increase and Counter Math.