Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 27 ยท Practice and Review

Prometheus and Grafana Cheat Sheet

One-page Prometheus and Grafana cheat sheet: config snippets, metric types, label matchers, common PromQL, recording and alert rules, burn rates and promtool commands.

This page is a quick reference for the things you look up most often while running Prometheus and Grafana: configuration skeletons, metric types, PromQL patterns, rule and alert shapes and promtool commands. It assumes you know the ideas; every section links to the lesson that explains them.

Prometheus and Grafana cannot run on this site, so every snippet is real and read-only. Two runnable stand-ins model label matchers and burn-rate math, so you can try the rules on sample data. They are models, not Prometheus.

Where to look first

Config prometheus.yml and rule files.
PromQL Selectors, rate, aggregation.
Alerts Rules, routing, burn rates.
promtool Check and test before deploy.

Check configs with promtool before every reload.

1 / 2

Configuration skeleton

YAML
# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: node
    file_sd_configs:
      - files: ["targets/nodes.json"]
        refresh_interval: 1m

Details: Installing Prometheus and Your First Scrape and How Prometheus Works.

Useful scrape job keys

Key Meaning
metrics_path Path to scrape (default /metrics)
scheme http or https
scrape_interval, scrape_timeout Per-job overrides (the timeout must not exceed the interval)
params URL query parameters, used by exporters such as blackbox
honor_labels Keep labels from the target instead of overwriting clashes
sample_limit, label_limit Fail the scrape if limits are exceeded (a cardinality guard)
relabel_configs Rewrite or drop targets before the scrape
metric_relabel_configs Rewrite or drop samples after the scrape
basic_auth, authorization, tls_config Credentials and TLS for the target

Relabeling actions

Action Effect
replace (default) Write the regex match of the source labels into target_label
keep / drop Keep or drop targets (or series) whose source labels match the regex
labelmap Copy labels whose names match the regex to new names
labeldrop / labelkeep Remove or keep labels by name regex
hashmod Hash the source labels into a number (sharding)

Regexes are fully anchored. See Service Discovery and Relabeling.

Metric types

Type Behavior Typical use Query with
Counter Only increases, resets on restart Requests, errors, bytes rate(), increase()
Gauge Goes up and down Queue length, memory, temperature Directly, avg_over_time(), deriv()
Histogram Bucket counters plus _sum and _count Latencies, sizes histogram_quantile()
Summary Client-side quantiles plus _sum and _count Quantiles that need no aggregation Read the quantile series

Naming: snake_case, base units, a unit suffix (_seconds, _bytes), _total on counters, no unbounded label values. See Counters, Gauges, Histograms and Summaries.

Text exposition format

TEXT
# HELP http_requests_total Total HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1027
http_requests_total{method="POST",status="500"} 3

Selectors and matchers

Matcher Meaning
= Equal
!= Not equal
=~ Matches regex (fully anchored)
!~ Does not match regex
PROMQL
http_requests_total{job="api", status=~"5..", method!="OPTIONS"}
http_requests_total{job="api"}[5m]              # range vector
http_requests_total{job="api"} offset 1h        # value one hour ago

A missing label equals the empty string, so {team=""} selects series without a team label. Try the matcher rules below, especially the anchoring. See PromQL Basics.

JS Label matchers on a few series

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Common PromQL patterns

Goal Query
Request rate per job sum by (job) (rate(http_requests_total[5m]))
Error ratio sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
p95 latency histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
Average latency sum(rate(x_duration_seconds_sum[5m])) / sum(rate(x_duration_seconds_count[5m]))
Targets down up == 0
Top 5 by rate topk(5, sum by (instance) (rate(http_requests_total[5m])))
CPU busy % (node_exporter) 100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
Memory used % 100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
Disk full in 4 h predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 4*3600) < 0
Missing metric absent(up{job="api"})
Number of series count({__name__=~".+"}) (expensive, use sparingly)
Fallback to 0 count(up == 0) or vector(0)

Aggregation and matching

PROMQL
sum by (job) (x)              # keep only job
sum without (instance) (x)    # drop only instance
avg, min, max, count, stddev, quantile(0.9, x), topk(3, x), bottomk(3, x)

a / on(instance) b                       # match on listed labels
a / ignoring(mode) b                     # match on all but these
a * on(instance) group_left(version) b   # many-to-one, copy "version"

Handy functions

Function Use
rate, irate, increase Counters (rate for alerts, irate for fast graphs)
delta, deriv, predict_linear Gauges
avg_over_time, max_over_time, quantile_over_time Summarize a window
histogram_quantile Quantiles from buckets
label_replace, label_join Edit labels in a query
clamp_min, clamp_max, round Limit or round values
time(), vector(), scalar() Convert between times and vectors
changes, resets Count changes or counter resets

Durations: ms, s, m, h, d, w, y (1h30m works). See rate, increase and Counter Math, Aggregation, Joins and Vector Matching and Quantiles, Histograms and SLI Queries.

Recording rules

YAML
groups:
  - name: http-recording
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_request_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
            / sum by (job) (rate(http_requests_total[5m]))

Naming is level:metric:operations. See Recording Rules and Query Performance.

Alert rules

YAML
groups:
  - name: alerts
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} of job {{ $labels.job }} is down"
          runbook_url: "https://runbooks.example.com/instance-down"

States: inactive, pending (true, waiting for for) and firing. Labels route; annotations describe. Templates use {{ $labels.name }} and {{ $value }}.

Alertmanager skeleton

YAML
route:
  receiver: team-chat
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="page"']
      receiver: on-call

receivers:
  - name: team-chat
    slack_configs:
      - channel: "#alerts"
        api_url_file: /etc/alertmanager/slack-url
  - name: on-call
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/pd-key

inhibit_rules:
  - source_matchers: ['severity="page"']
    target_matchers: ['severity="warning"']
    equal: [alertname, job]

See Alerting Rules and Alertmanager.

Multi-window burn rates (99.9% SLO, 30 days)

Burn rate Long window Short window Budget used in the long window
14.4 1 h 5 m 2%
6 6 h 30 m 5%
1 3 d 6 h 10%

The error-ratio threshold is the burn rate times the error budget (0.001). The stand-in checks the table and lets you try other objectives.

JS Burn-rate thresholds

Press Run (or Ctrl + Enter) to see the console output here.

No output.

promtool

Command Purpose
promtool check config prometheus.yml Validate the main config and the rule files it lists
promtool check rules rules/*.yml Validate rule files
promtool test rules tests/alerts_test.yml Run unit tests against synthetic series
promtool check metrics < /metrics.txt Lint exposition-format text (naming, types)
promtool query instant http://localhost:9090 'up' Run an instant query
promtool query range --start=... --end=... http://localhost:9090 'rate(x[5m])' Run a range query
promtool tsdb list /data List blocks in a data directory
curl -X POST localhost:9090/-/reload Reload the config (needs --web.enable-lifecycle)

Rule test file shape:

YAML
rule_files: ["../rules/alerts.yml"]
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'up{job="api", instance="a:8080"}'
        values: '1 1 0 0 0 0 0 0'
    alert_rule_test:
      - eval_time: 7m
        alertname: InstanceDown
        exp_alerts:
          - exp_labels: { severity: page, job: api, instance: "a:8080" }

Useful flags and endpoints

Item Meaning
--config.file Path of the config
--storage.tsdb.path, --storage.tsdb.retention.time Data directory and retention (default 15d)
--web.enable-lifecycle Enables the /-/reload endpoint
/targets, /service-discovery, /rules, /alerts Debug pages
/-/healthy, /-/ready Health checks
/api/v1/query, /api/v1/query_range HTTP query API

Grafana in one screen

Task How
Add Prometheus Connections, Data sources, Prometheus, URL http://prometheus:9090
Provision a data source A YAML file in provisioning/datasources/ with type: prometheus and url
Provision dashboards A provider in provisioning/dashboards/ pointing to a folder of JSON files
Variable for instances Query variable: label_values(up, instance)
Use a variable rate(x{instance=~"$instance"}[$__rate_interval])
Units Set the panel unit (seconds, bytes, percent 0-1) instead of scaling in the query

$__rate_interval is Grafana's safe window for rate(): at least four scrape intervals. See Grafana: Data Sources and Dashboards and Designing Dashboards That Help.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which selector matches series whose status label is exactly 503, 504 or 502?

    Regex matchers are fully anchored, so 50[2-4] matches exactly the three codes. A bare 5 would match only a status of 5.
  2. 2. What is the error-ratio threshold for a burn rate of 6 with a 99.9% SLO?

    The threshold is the burn rate times the error budget: 6 times 0.001 is 0.006, or 0.6%.
  3. 3. Which tool runs unit tests for alert rules with synthetic input series?

    check rules only validates syntax. test rules evaluates rules against input series and asserts the resulting alerts.
  4. 4. Where do you drop series by metric name after the scrape?

    metric_relabel_configs runs on scraped samples, so matching name belongs there.

Summary

  • Keep a config skeleton, the four metric types and the matcher rules (anchored regexes, missing label equals empty string) in your head.
  • Reuse the PromQL patterns: rate inside sum, le kept for quantiles, predict_linear for disks.
  • Recording rules follow level:metric:operations; alerts use for, labels for routing and annotations for people.
  • Thresholds for burn-rate alerts are the burn rate times the error budget.
  • Validate with promtool check and promtool test before every reload.

You have reached the end of the tutorial. Revisit any lesson from the introduction, or repeat the exercises.