Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 27 ยท Prometheus Basics

Counters, Gauges, Histograms and Summaries

Learn the four Prometheus metric types, when to use each, how counter resets and histogram buckets work, and how quantiles are estimated.

Prometheus defines four kinds of metric. Choosing the right one matters, because the type decides which queries make sense later. A counter is for things that only grow, a gauge for things that move up and down, and a histogram or summary for the shape of a distribution such as request durations.

This lesson explains each type, what it looks like on /metrics, when to use it and its pitfalls. A runnable JavaScript implementation of all four shows what happens inside the client libraries. Note that the type is only a hint in the exposition format. The server stores everything as plain samples, and the type matters through the functions you choose in PromQL.

Four metric types

Counter Only goes up. Query its rate.
Gauge Goes up and down. Read it directly.
Histogram Counts per bucket. Estimate quantiles.
Summary Quantiles computed in the app.

Not sure? Counter for events, gauge for levels, histogram for durations.

1 / 2

Counter

A counter is a cumulative number that only goes up (or resets to zero when the process restarts). Use it for anything you count: requests served, errors, bytes sent, tasks completed. By convention the name ends in _total.

TEXT
# TYPE http_requests_total counter
http_requests_total{method="GET",code="200"} 1027

The raw value of a counter is rarely useful ("1027 requests since the process started"), so you almost always query its rate of change, such as requests per second, with functions like rate() (rate, increase and Counter Math). Two reasons counters are so good:

  • If a scrape is missed, no information is lost: the next value still contains the total, so the rate can be computed across the gap.
  • A restart is detectable: when the value drops, Prometheus knows the counter was reset and corrects for it.

Never use a counter for something that can decrease. "Number of logged-in users" is a gauge.

Gauge

A gauge is a value that can go up and down: memory in use, queue length, temperature, number of running jobs, a configuration flag as 0 or 1. Query it directly, or use functions such as avg_over_time(), max_over_time() and delta().

TEXT
# TYPE queue_length gauge
queue_length{queue="emails"} 17

A gauge only shows the value at scrape time. If it spikes and falls back between scrapes, you never see the spike. When you care about peaks, expose a counter or histogram of the events instead, or a gauge holding the maximum over a short window.

Histogram

A histogram records how many observations fell into each range. You choose the bucket boundaries in advance, for example 0.05, 0.1, 0.25, 0.5 and 1 second. The client library exposes three kinds of series per histogram:

TEXT
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 3
http_request_duration_seconds_bucket{le="0.1"} 7
http_request_duration_seconds_bucket{le="0.25"} 11
http_request_duration_seconds_bucket{le="+Inf"} 20
http_request_duration_seconds_sum 8.7
http_request_duration_seconds_count 20
  • _bucket{le="X"} is a counter of observations less than or equal to X. Buckets are cumulative: le="0.25" includes everything in the smaller buckets. The last bucket, +Inf, equals the total count.
  • _sum is the sum of all observed values and _count is the number of observations, both counters. The average is _sum / _count.

Because histograms are made of counters, they work with rate(), and (the important part) they can be aggregated across instances: add up the buckets from 50 servers and you get the overall distribution. The query function histogram_quantile() then estimates quantiles (Quantiles, Histograms and SLI Queries).

The cost is that quantiles are estimates. Prometheus knows only how many observations were in each bucket, so it assumes they are spread evenly inside the bucket and interpolates. The estimate can never be more precise than the bucket width, so choose buckets around the latencies you care about, such as your target and your timeout. Every bucket is its own series too, so many buckets multiplied by many labels adds to cardinality.

Recent Prometheus versions also support native histograms, which use automatically sized, much finer buckets stored in a single series. They are opt-in and need support in both the client library and the server, so check the documentation of your version before relying on them. Everything in this tutorial uses classic bucket histograms.

Summary (the metric type)

A summary also tracks _sum and _count, but computes quantiles inside the application and exposes them directly:

TEXT
# TYPE rpc_duration_seconds summary
rpc_duration_seconds{quantile="0.5"} 0.2
rpc_duration_seconds{quantile="0.95"} 1.4
rpc_duration_seconds_sum 8.7
rpc_duration_seconds_count 20

The quantiles are accurate for that one process. The trouble is that quantiles cannot be averaged or added: the p95 of the whole fleet is not the average of each server's p95. If you run more than one instance and want a combined percentile, use a histogram. Choose a summary only when you need accurate quantiles for a single process, and know which quantiles you want ahead of time.

All four types in code

The stand-in below implements the four types the way a client library does, feeds 20 request durations through them, and prints what would be exposed. The histogram's quantile method uses the same idea as histogram_quantile(): find the bucket that contains the wanted rank, then interpolate. The summary here keeps every value for simplicity, whereas real libraries use streaming algorithms that approximate the quantiles in constant memory.

JS Implement counter, gauge, histogram and summary

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Compare the two ways of finding percentiles. The histogram estimates p50 as 0.212 s, close to the exact 0.2 s, because the median lands in a narrow bucket. For p95 it says 1.000 s, while the exact value is 1.4 s. The 95th observation lies beyond the biggest finite bucket (1 s), so the estimate can only report that bound. That is the price of buckets: add a 2.5 or 5 bucket, or move the top bucket above your timeout, if you need to see the slow tail. The counter and gauge behave as promised, and a negative increment is rejected, like in real client libraries.

Counter resets

A counter lives in the process's memory, so a restart sets it back to zero. Prometheus functions treat any decrease as a reset and continue from the new value. The stand-in below shows why that matters: a naive "last minus first" calculation gets the restart wrong, while a reset-aware increase gives the right answer.

JS Why counter resets need special handling

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The naive difference is -10, which would claim the service handled a negative number of requests. The reset-aware version adds up each step and, when the value falls, assumes the counter restarted at zero and counts the new value as growth, giving 210. That is the idea behind rate() and increase(), which also correct for resets.

Choosing the right type

You want to measure Use
Requests, errors, bytes, jobs finished Counter
Memory, queue length, temperature, in-flight requests Gauge
Request duration or size, aggregated over many servers Histogram
Accurate quantiles of one process Summary
When something last happened Gauge holding a Unix timestamp

What is real and what is the stand-in

The four types, their exposition lines, cumulative buckets, le="+Inf" and the reset behavior are real. The classes above are teaching versions: real client libraries are thread-safe, support labels and use efficient streaming algorithms for summaries, and histogram_quantile() works on rates of bucket counters from the database, not on in-process arrays. Libraries are the topic of Instrumenting Your Code.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which type should count the number of failed login attempts?

    It only increases, so it is a counter. You then query its rate.
  2. 2. Which type fits the number of items currently in a queue?

    A queue length rises and falls, which is what a gauge is for.
  3. 3. Why can histograms be combined across many servers, while summary quantiles cannot?

    Adding bucket counters from every instance gives the fleet-wide distribution, but quantiles from separate processes cannot be added or averaged.
  4. 4. A histogram has buckets up to 1 second, but 5% of requests take longer. What happens to your p99 estimate?

    Observations beyond the last finite bucket are only counted in +Inf, so the estimate cannot go above the highest finite bound.
  5. 5. What happens to a counter when the application restarts?

    The value lives in process memory, so it restarts from zero. Prometheus treats a decrease as a reset.

Summary

  • A counter only increases; query its rate. A gauge moves up and down; read it directly.
  • A histogram exposes cumulative _bucket{le} counters plus _sum and _count; quantiles are estimates limited by the bucket width.
  • A summary exposes precomputed quantiles for one process; they cannot be aggregated across instances.
  • Counter resets are detected as decreases and corrected by rate() and increase().
  • Choose buckets around your latency targets and timeouts, and prefer histograms when you run several instances.

Next: Instrumenting Your Code.