Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 27 ยท Practice and Review

Exercises with Solutions

Twelve graded Prometheus exercises on PromQL, instrumentation, alert rules and relabeling, each with a worked answer and runnable stand-ins for the math.

You learn a query language by writing queries, and you learn alerting by deciding what deserves to wake someone up. This page gives you 12 exercises, graded from easy to hard, across the four skills that matter most day to day: PromQL, instrumentation, alert rules and relabeling. Every task has a worked solution right below it.

Use it like this: read the task, write your own answer on paper or in the expression browser of a real Prometheus, and only then open the solution. Different but correct answers are normal in PromQL, so compare the idea, not the characters.

Real versus stand-in. Prometheus, Alertmanager and Grafana cannot run on this site, so every PromQL query, YAML file and client snippet below is real and read-only. The runnable examples are small stand-ins written in Python and JavaScript that reproduce the arithmetic and logic behind several solutions: counter-reset-aware rate, quantile interpolation from buckets, relabeling and the pending-to-firing alert state machine. They are simplified models, not Prometheus. They leave out details such as extrapolation, so their numbers can differ slightly from a real server.

Four skills, twelve tasks

PromQL Rates, ratios, quantiles, predictions.
Instrumentation Types, names and safe labels.
Alert rules Symptoms, for, annotations.
Relabeling Keep, rewrite and drop.

Levels are Easy, Medium and Hard.

1 / 2

Part 1: PromQL (exercises 1 to 5)

Review: PromQL Basics, rate, increase and Counter Math, Aggregation, Joins and Vector Matching and Quantiles, Histograms and SLI Queries.

Exercise 1 (Easy): requests per second per job

The counter http_requests_total has the labels job, instance, method and status. Show the request rate in requests per second for each job, averaged over the last 5 minutes.

Solution

PROMQL
sum by (job) (rate(http_requests_total[5m]))

rate() works on one series at a time and handles counter resets, so it goes inside the aggregation. sum by (job) then adds the per-series rates and keeps only the job label. Writing rate(sum(...)) is the classic mistake: after summing, a reset in one instance looks like a drop in the total and cannot be repaired.

Exercise 2 (Easy): which targets are down

List every scrape target that Prometheus could not reach in its latest scrape, and then count them.

Solution

PROMQL
up == 0
PROMQL
count(up == 0)

up is 1 when the last scrape succeeded and 0 when it failed. The comparison without bool is a filter: it keeps only the series where the condition holds. When no target is down, count(...) returns an empty result, not 0. If you need a number for a dashboard, use count(up == 0) or vector(0).

Exercise 3 (Medium): the error ratio

Compute the fraction of all requests that failed with a status code from 500 to 599, as a ratio between 0 and 1, across the whole service.

Solution

PROMQL
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

The regular expression 5.. is anchored automatically, so it matches exactly three characters starting with 5. Both sides are aggregated to a single series with no labels, so the division matches them without any on() clause. If you wanted a ratio per job, you would write sum by (job) on both sides.

The first stand-in below re-creates the math: a counter with a reset, a rate per status, and the ratio. Run it, then change the reset position in ok_values.

Python Counter-reset-aware rate and error ratio

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise 4 (Medium): the 95th percentile latency

The histogram http_request_duration_seconds exposes the series _bucket (with the label le), _sum and _count. Show the 95th percentile request latency over the last 5 minutes for the whole service.

Solution

PROMQL
histogram_quantile(
  0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

Two rules matter here. First, rate() on the buckets turns cumulative counters into per-second rates for the window. Second, the aggregation must keep le; if you drop it, there is nothing left to interpolate. To keep a per-job quantile, write sum by (job, le).

The result is an estimate. Prometheus assumes observations are spread evenly inside the bucket that contains the quantile, so wide buckets give coarse answers. The second stand-in does the same interpolation on cumulative bucket counts.

Python histogram_quantile from cumulative buckets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise 5 (Hard): will the disk fill up?

Using node_filesystem_avail_bytes from node_exporter, write an expression that is true for every real filesystem that is predicted to run out of space within the next 4 hours, based on the trend of the last 6 hours. Ignore tmpfs and overlay filesystems.

Solution

PROMQL
predict_linear(
  node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h],
  4 * 3600
) < 0

predict_linear(v[6h], 14400) fits a straight line through the last 6 hours of each series and returns where the line will be 14400 seconds from now. A negative number of free bytes means "full". The comparison filters out every healthy filesystem. A 6-hour window smooths out short bursts; a 15-minute window would fire on every large log rotation. See Project: Monitor a Fleet of Servers for the full setup.

Part 2: Instrumentation (exercises 6 to 8)

Review: Metrics, Labels and Time Series, Counters, Gauges, Histograms and Summaries and Instrumenting Your Code.

Exercise 6 (Easy): pick the metric type

Choose counter, gauge, histogram or summary for each measurement, and say why.

  1. Total orders placed since the process started.
  2. Orders currently waiting in a queue.
  3. How long each payment call takes, so you can compute percentiles across many servers.
  4. Seconds since the last successful backup.

Solution

  1. Counter. It only goes up, and you query it with rate() or increase().
  2. Gauge. The value moves both ways, and you read it directly.
  3. Histogram. Buckets can be summed across instances before the quantile is computed. A summary's quantiles are calculated inside each process and cannot be averaged meaningfully.
  4. Gauge. Either export the timestamp of the last success and compute time() - metric in the query, or export the age. The timestamp is more robust, because it stays correct between scrapes.

Exercise 7 (Medium): fix the names and labels

A teammate wrote these metrics. Rewrite each to follow the conventions, and explain what is wrong.

TEXT
requestLatencyMs{user_id="48213", path="/orders/48213"}
total_cache_hits
memory_usage_percent_used

Solution

TEXT
http_request_duration_seconds{method="GET", route="/orders/:id"}
cache_hits_total
process_resident_memory_bytes
  • Names use snake_case, base units (seconds and bytes, never milliseconds or percent) and carry a unit suffix.
  • Counters end in _total, and the total_ prefix is wrong.
  • user_id and a raw path with the ID inside create one new series for every user and every order. That is a cardinality explosion. Label values must come from a small, bounded set, so use the route template.
  • Percentages are a derived value: export used and total bytes, and compute the ratio in PromQL.

Exercise 8 (Hard): instrument a handler

Using the Python client library prometheus_client, instrument a checkout handler: count requests by outcome and measure the duration with sensible buckets. Expose the metrics on port 8000.

Solution

Python
import time
from prometheus_client import Counter, Histogram, start_http_server

CHECKOUTS = Counter(
    "checkout_requests_total",
    "Checkout requests by outcome.",
    ["outcome"],
)
CHECKOUT_SECONDS = Histogram(
    "checkout_duration_seconds",
    "Time spent processing a checkout.",
    buckets=(0.05, 0.1, 0.25, 0.5, 1, 2.5, 5),
)

def handle_checkout(order):
    started = time.perf_counter()
    try:
        process(order)                      # your business logic
        CHECKOUTS.labels(outcome="success").inc()
    except PaymentDeclined:
        CHECKOUTS.labels(outcome="declined").inc()
        raise
    except Exception:
        CHECKOUTS.labels(outcome="error").inc()
        raise
    finally:
        CHECKOUT_SECONDS.observe(time.perf_counter() - started)

if __name__ == "__main__":
    start_http_server(8000)   # metrics at http://localhost:8000/metrics
    serve_forever()

The outcome label has three possible values, so it is bounded. The buckets surround the latency you care about (for example a 500 ms objective), because quantile accuracy depends on how close the buckets are to the target. The finally block makes sure that failed requests are timed too; leaving them out hides your slowest cases. Initialize known label values at start-up (CHECKOUTS.labels(outcome="declined") without .inc()) if you want the series to exist with value 0 before the first event.

Part 3: Alert rules (exercises 9 to 10)

Review: Alerting Rules and Alertmanager and Project: Alerts with Burn Rates and Routing.

Exercise 9 (Medium): alert on the error ratio

Write an alert rule that fires when more than 5% of a job's requests fail for 10 minutes in a row. Give it a severity, a summary and a runbook link.

Solution

YAML
groups:
  - name: web-alerts
    rules:
      - alert: HighErrorRatio
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
            /
          sum by (job) (rate(http_requests_total[5m]))
            > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "More than 5% of requests fail on {{ $labels.job }}"
          runbook_url: "https://runbooks.example.com/high-error-ratio"

This alerts on a symptom that users feel, not on a cause such as CPU load. for: 10m means the condition must hold on every evaluation for ten minutes; the alert stays pending until then, which filters out short blips. severity is a label because Alertmanager routes on labels. The summary is an annotation because it is text for humans and must not split alerts into separate groups.

Exercise 10 (Hard): predict the alert timeline

The rule above evaluates every 15 seconds with for: 1m (shortened for the exercise). The condition is true at 0 s, 15 s and 30 s, false at 45 s, then true from 60 s onward. At which evaluation does the alert fire, and what happens at 45 s?

Solution

The condition becomes true at 0 s, so the alert is pending from 0 s. It would fire once it has been true for 60 seconds, but at 45 s the condition is false. The pending alert is dropped and the timer resets. The next true value at 60 s starts a new pending period, so the alert fires at 120 s, when the condition has held for a full minute. The stand-in below runs this exact timeline, so you can try other sequences.

JS Pending and firing state machine

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Part 4: Relabeling (exercises 11 to 12)

Review: Service Discovery and Relabeling.

Exercise 11 (Medium): choose and tidy targets

A file-based service discovery source lists hosts with the meta labels __meta_env and __meta_team. Write relabel_configs that scrape only production hosts, and put the team into a label called team. Then drop every series whose name starts with go_.

Solution

YAML
scrape_configs:
  - job_name: apps
    file_sd_configs:
      - files: ["targets/*.json"]
    relabel_configs:
      - source_labels: [__meta_env]
        regex: prod
        action: keep
      - source_labels: [__meta_team]
        target_label: team
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*"
        action: drop

relabel_configs runs before the scrape and works on targets. keep removes every target whose value does not match the (fully anchored) regex. With no action, the default is replace, which copies the source value into team. metric_relabel_configs runs after the scrape, on each sample, so it is the right place to drop series by name. Labels beginning with __meta_ are discarded after relabeling finishes, which is why the team copy is needed.

Exercise 12 (Hard): relabel targets in code

Implement keep, replace and drop on a list of targets and verify that your answer to exercise 11 selects the right hosts.

Solution

The stand-in below applies rules in order to three targets and then filters a few metric names. Notice that regular expressions are anchored on both ends, exactly like in Prometheus.

JS Applying relabel rules

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Where to go next. If an exercise felt hard, reread the linked lesson and rerun the stand-in with different numbers. Then install Prometheus locally (Installing Prometheus and Your First Scrape) and run each query against real data, because the expression browser is the fastest teacher.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does rate() go inside sum() and not around it?

    rate() detects a reset by seeing a series drop. After summing, one instance restarting only makes the total look smaller, and the reset cannot be repaired.
  2. 2. What does an alert do when its condition turns false while it is pending?

    A pending alert that stops matching is dropped. If the condition returns later, a new pending period begins from zero.
  3. 3. Which label is a cardinality risk?

    Every distinct user_id creates a new time series. Label values must be drawn from a small, bounded set.
  4. 4. In which block do you drop series by metric name after a scrape?

    metric_relabel_configs runs on every scraped sample, so name matching belongs there. relabel_configs runs earlier and works on targets.

Summary

  • Put rate() inside the aggregation, keep le when computing quantiles and prefer ratios of rates for error and availability queries.
  • Choose types by behavior: counters for totals, gauges for current values, histograms when percentiles must be aggregated across instances.
  • Keep label values bounded, use base units and _total suffixes, and measure failed requests as well as successful ones.
  • Alert on symptoms, use for to filter blips, and remember that a pending alert resets when its condition turns false.
  • relabel_configs shapes targets before the scrape; metric_relabel_configs filters series after it.

Next, test yourself with the Interview Questions.