Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 27 ยท Prometheus Basics

Metrics, Labels and Time Series

Learn the Prometheus data model: metric names, labels, samples, time series, cardinality, naming conventions and the text exposition format.

Everything in Prometheus is built on one simple idea: a stream of numbers, each with a timestamp, identified by a name and a set of labels. Once you understand how that identity works, queries, dashboards and alert rules become much easier to read. You also learn to avoid the most common way to break a Prometheus server, which is creating too many distinct series.

This lesson covers the data model, naming rules and the plain-text format that applications use to expose their values.

Anatomy of a sample

Metric name What is measured.
Labels Dimensions that split it.
Value A 64-bit float.
Timestamp When it was scraped.

Name plus labels identifies one time series.

1 / 2

Time series and samples

A sample is one value at one moment: a 64-bit floating-point number and a millisecond timestamp. A time series is the sequence of samples over time that belongs to one identity. The identity is the metric name plus a set of labels, which are key-value pairs.

TEXT
http_requests_total{method="GET", code="200"}

Here http_requests_total is the metric name, and method="GET" and code="200" are labels. Change any label and you have a different series: http_requests_total{method="GET", code="500"} counts something else and is stored separately. Labels let you slice one metric many ways, such as total requests, requests per method or only failures, without inventing a new metric name for each.

A few rules to remember:

  • A metric name matches [a-zA-Z_:][a-zA-Z0-9_:]*. Names with colons are reserved for recording rules.
  • A label name matches [a-zA-Z_][a-zA-Z0-9_]*. Names starting with __ (two underscores) are reserved for Prometheus internals.
  • A label value can be any text. A label with an empty value is treated as if the label did not exist.
  • Prometheus adds the job and instance labels to everything it scrapes, as you saw in How Prometheus Works.

Naming conventions

Prometheus has strong naming conventions. Following them makes metrics from different teams look familiar and prevents unit confusion.

  • Use snake_case and start with a prefix naming the application or subsystem, for example myapp_orders_created_total.
  • Use base units: seconds (not milliseconds), bytes (not megabytes), meters, ratios from 0 to 1 (not percentages). Put the unit in the name as a suffix: http_request_duration_seconds, process_resident_memory_bytes.
  • A counter name should end with _total.
  • Do not put label names into the metric name. http_requests_total{method="GET"} is right, http_get_requests_total is not.
  • A metric name should describe one thing. If summing or averaging over all label values makes no sense, it should probably be two metrics.

The text exposition format

Applications expose metrics on /metrics in a line-based text format that is easy to read and easy to produce, even by hand. Here is a real-looking response:

TEXT
# HELP http_requests_total Total HTTP requests handled.
# TYPE http_requests_total counter
http_requests_total{method="GET",code="200"} 1027
http_requests_total{method="GET",code="500"} 3
http_requests_total{method="POST",code="200"} 412
# HELP memory_usage_bytes Resident memory in bytes.
# TYPE memory_usage_bytes gauge
memory_usage_bytes 52428800
  • # HELP lines are human-readable descriptions.
  • # TYPE lines declare the metric type: counter, gauge, histogram, summary or untyped (see Counters, Gauges, Histograms and Summaries).
  • Every other line is name{labels} value. A metric without labels omits the braces.
  • Inside quotes, backslash, double quote and newline are escaped with a backslash.
  • A sample line may end with an optional millisecond timestamp, which is rarely used, because Prometheus normally stamps data with the scrape time.

Prometheus can also scrape a richer format called OpenMetrics, which adds features such as exemplars and an # EOF marker, but the plain text above is always accepted and is what most client libraries produce.

Parsing exposition text

The stand-in below is a small parser for this format. It reads the page above (plus a gauge with a space inside a label value), prints each sample with its type and counts the distinct series. It ignores the optional timestamp and escape sequences, which the real parser handles. The code runs on Piston, a server-side runner, the same way other Node lessons do.

Node.js Parse the text exposition format

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Notice that http_requests_total produced three samples, because it has three label combinations, and each is one series. Together with the other two metrics, the page contains five series. The parser attaches the type to each sample by looking up the metric name, just as Prometheus remembers metadata per metric.

Cardinality

Cardinality is the number of distinct time series. Each unique combination of label values is a separate series, so cardinality is the product of the number of values for each label. With 4 methods and 6 status codes you get 24 series per metric, which is fine. Add a route label with 25 values and you have 600. Add a user_id label with 50,000 users and you have 30 million.

Every series costs memory and CPU for the whole time it is active, and Prometheus slows down and can run out of memory when cardinality grows too large. The rules of thumb:

  • Use labels for small, bounded sets of values: status code, method, route template (/users/:id, not the real URL), region, version.
  • Never use unbounded values as labels: user ids, session ids, email addresses, full URLs, timestamps, error message text.
  • If you need per-request detail, use logs or traces, which are made for it (What Monitoring Is and Why It Matters).

Seeing the explosion

The next stand-in multiplies label value counts and applies the naming conventions as a simple linter. The 3 KB per series figure is only a rough assumption to show the order of magnitude: real memory use depends on the version and the labels.

Node.js Cardinality arithmetic and a naming linter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

One extra label turned 600 series into 30 million. The linter then flags the usual naming mistakes: camel case, milliseconds instead of seconds, and names with no unit. (The real convention is looser than this toy check: for instance a gauge such as cache_size is acceptable when the unit is obvious, though cache_entries or cache_size_bytes is clearer.) You will measure real cardinality later in Security, Cardinality and Common Pitfalls.

What is real and what is the stand-in

The data model, naming rules, label rules and text format described above are real. The parser and linter are simplified stand-ins: the real parser follows the full specification (escapes, timestamps, OpenMetrics, histogram lines), and the real server stores samples in a compressed time series database rather than in plain objects.

A label that changes constantly (a request id, a timestamp) creates a brand-new series on every scrape. This is the most common cause of Prometheus running out of memory.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What uniquely identifies a time series?

    Changing the metric name or any label value creates a different series.
  2. 2. A metric has labels with 3, 4 and 5 distinct values. How many series can it have at most?

    Cardinality multiplies: 3 x 4 x 5 = 60 possible combinations.
  3. 3. Which is a good metric name by Prometheus conventions?

    Use snake_case and base units, with the unit as a suffix: seconds, not milliseconds.
  4. 4. Which should NOT be used as a label?

    Unbounded values create one series per value, which can overwhelm Prometheus.
  5. 5. What does a # TYPE line declare in the exposition format?

    TYPE gives the type of the named metric, and # HELP gives its description.

Summary

  • A time series is a stream of timestamped samples identified by a metric name plus labels.
  • Use snake_case, base units and unit suffixes (_seconds, _bytes, _total for counters).
  • The text exposition format has # HELP, # TYPE and name{labels} value lines.
  • Cardinality is the product of distinct label values, so keep label values bounded and small.
  • Per-request detail belongs in logs and traces, not in labels.

Next: Counters, Gauges, Histograms and Summaries.