Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 27 ยท Instrumenting Applications

Instrumenting Your Code

Add Prometheus metrics to your own code with client libraries for Python, Go, Java and Node.js, expose /metrics, and avoid label and naming mistakes.

Exporters give you metrics about machines and databases, but nobody can tell you how many checkouts failed in your shop, or how long your report query takes. Only your own code knows that. Instrumentation means adding a few lines to the application so that it counts and times the things you care about, and publishes the numbers where Prometheus can scrape them.

You do this with a client library: a small package that keeps metric values in memory, formats them in the text exposition format from Metrics, Labels and Time Series, and serves them over HTTP. Official libraries exist for Go, Java, Python and Ruby, and there are well-maintained community ones for Node.js, .NET, PHP, Rust and more. They all work the same way, so once you know one, you know them all.

The real libraries need a running Prometheus and an installed package, so the library code below is read-only. At the end you will build a tiny registry of your own that renders a real /metrics page, which shows exactly what the libraries do for you.

The shape of every client library

Four steps, every language

Registry One place that owns all metrics.
Metrics Counters, gauges, histograms you declare once.
Update Call inc, set or observe where things happen.
Expose Serve the registry as text at /metrics.
1 / 2

A registry is the object that holds every metric. Libraries create a default one, so most programs never touch it directly. You declare each metric once, usually as a module-level variable. Then you update it from wherever the event happens. Finally you expose the registry over HTTP, either with a built-in helper server or by adding a route to your existing web framework.

Python

The official prometheus_client package starts a tiny extra HTTP server for you:

Python
from prometheus_client import Counter, Gauge, Histogram, start_http_server
import random, time

ORDERS = Counter(
    "shop_orders_total", "Orders placed.", ["payment_method"]
)
QUEUE = Gauge("shop_queue_length", "Orders waiting for packing.")
REPORT_SECONDS = Histogram(
    "shop_report_duration_seconds",
    "Time to build the daily report.",
    buckets=(0.1, 0.5, 1, 2.5, 5, 10),
)

start_http_server(8000)           # serves http://localhost:8000/metrics

while True:
    ORDERS.labels(payment_method="card").inc()
    QUEUE.set(random.randint(0, 20))
    with REPORT_SECONDS.time():   # observes the elapsed seconds
        time.sleep(0.2)

Counter, Gauge and Histogram are classes; labels(...) picks one child series out of the metric. The .time() context manager (also usable as a decorator) measures a block and calls observe for you. Process metrics such as CPU time and open file descriptors are registered automatically on Linux.

Go

In Go you usually mount the handler on your own server. The promauto package registers each metric on creation so you cannot forget:

Go
package main

import (
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
	requests = promauto.NewCounterVec(prometheus.CounterOpts{
		Name: "http_requests_total",
		Help: "HTTP requests by method and status code.",
	}, []string{"method", "code"})

	latency = promauto.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "http_request_duration_seconds",
		Help:    "Request latency in seconds.",
		Buckets: prometheus.DefBuckets,
	}, []string{"route"})
)

func main() {
	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":8080", nil)
}

The default registry in Go also contains Go runtime metrics (go_goroutines, go_memstats_alloc_bytes) and process metrics, so a new service already has a dozen useful series before you add one of your own.

Java

The current Java client (the 1.x line) builds metrics with builders and offers a small built-in server. Many Spring Boot applications instead use Micrometer, a facade that can publish to Prometheus among other systems, but the idea is the same:

Java
import io.prometheus.metrics.core.metrics.Counter;
import io.prometheus.metrics.exporter.httpserver.HTTPServer;
import io.prometheus.metrics.instrumentation.jvm.JvmMetrics;

public class App {
    public static void main(String[] args) throws Exception {
        JvmMetrics.builder().register();            // heap, GC, threads...

        Counter jobs = Counter.builder()
            .name("worker_jobs_total")
            .help("Jobs processed.")
            .labelNames("outcome")
            .register();

        HTTPServer.builder().port(9400).buildAndStart();

        jobs.labelValues("success").inc();
    }
}

Check the exact package names against the version you install; the 0.x series used different ones.

Node.js

The popular community package is prom-client. Its registry renders asynchronously, so the route handler awaits it:

JavaScript
import express from "express";
import client from "prom-client";

client.collectDefaultMetrics();                       // event loop, memory, GC

const httpDuration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "Request latency in seconds.",
  labelNames: ["method", "route", "status"],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5],
});

const app = express();

app.use((req, res, next) => {
  const end = httpDuration.startTimer();
  res.on("finish", () => {
    end({ method: req.method, route: req.route?.path ?? "unmatched", status: res.statusCode });
  });
  next();
});

app.get("/metrics", async (req, res) => {
  res.set("Content-Type", client.register.contentType);
  res.end(await client.register.metrics());
});

app.listen(3000);

Notice the pattern in the middleware: start a timer when the request arrives, stop it when the response finishes, and label by the route template (/users/:id), not by the actual URL. The next sections explain why that matters.

Instrument with middleware, not by hand

You rarely need to sprinkle inc() calls through every handler. Web frameworks have middleware or interceptors that see every request, so one piece of code can record all of the RED signals (rate, errors, duration) from RED, USE and the Golden Signals:

  • a counter of requests labelled by method, route template and status code gives you rate and errors;
  • a histogram of durations labelled by route gives you duration;
  • optionally a gauge of in-flight requests shows saturation.

Add hand-written metrics for business events (orders, sign-ups, jobs, cache hits) and for slow external calls (database queries, calls to other services). Those are the numbers an engineer will want at 3 a.m.

Labels to avoid

Every distinct combination of label values is a separate time series that Prometheus stores and indexes forever (until retention removes it). A label whose values are unbounded multiplies the series count without limit.

Bad label Why it hurts Better
user_id, email One series per user Count users in logs or a database
raw URL path /users/8431 One series per id Route template /users/:id
request_id, trace_id New series for every request Use tracing; exemplars link traces to histograms
full error message Messages contain ids and timestamps A short error type such as timeout
timestamps, IP addresses Unbounded Leave them out

A useful rule of thumb: a label should have at most a few dozen values, and you should be able to list them all in advance. If you cannot, it belongs in logs or traces, not in metric labels. When you do need several label dimensions, remember that series multiply: 5 methods x 40 routes x 8 status codes is already 1,600 series per histogram bucket.

Naming and units

Consistent names make dashboards and queries predictable:

  • Use snake_case with a prefix for the application or domain: shop_orders_total.
  • Counters end in _total. Without it the name does not say that the value only grows.
  • Put the unit in the name, in base units: _seconds (never milliseconds), _bytes (never kilobytes), _ratio for 0 to 1 values. Units in the name make a mistake like adding seconds to milliseconds impossible to miss.
  • The name describes what is measured, not how it is stored. Do not repeat label names in it (orders_by_payment_method is wrong; use a payment_method label).
  • Never put the same quantity under two names or a different type. A sum over a metric must make sense across all of its label values.

Also initialise the label combinations you know about, so that a counter exists at 0 before the first event. Otherwise rate(...) over a series that appears only after its first increment loses that first increase, and absent()-style alerts cannot tell "no errors" from "no data".

Build a registry yourself

A client library is less magic than it looks. This stand-in is not the real library: it is a small registry with a counter, a gauge and a histogram that renders the text exposition format, written in plain JavaScript so it runs on Piston. A fake web service updates it with a fixed list of requests, and the final call prints the page a scrape would receive.

Node.js A tiny registry that renders /metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the output against the real format. Each metric has a # HELP and # TYPE line, counters carry their label sets, and the histogram shows cumulative buckets (le="0.5" counts everything that was at most 0.5 seconds, including the faster requests) plus _sum and _count. A scrape is just an HTTP GET that returns this text; the Prometheus server stores one sample per line.

Why label choice matters, in numbers

The next stand-in counts how many series a counter creates when the middleware labels by the raw path versus the route template. It is a stand-in for the real cost: every series is a row Prometheus must keep in memory and on disk.

JS Counting series: raw path or route template

Press Run (or Ctrl + Enter) to see the console output here.

No output.

801 requests produced 801 series with the raw path, and a busy site produces millions of different URLs. The route template keeps it at three. Because the histogram adds a series per bucket plus _sum and _count, the damage is multiplied again.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where should you create a Prometheus metric object in your program?

    Metrics hold state in memory and are registered in a registry. They are declared once and updated from many places. Creating one per request would try to register the same name repeatedly or lose the accumulated values.
  2. 2. Which label would be the most dangerous to put on a request counter?

    A user id has as many values as you have users, so every user creates new series for ever. The others have a small, known set of values.
  3. 3. A histogram measures how long an operation takes. Which metric name follows the naming conventions?

    Names use snake_case, describe what is measured and carry the base unit (seconds). The suffix _total is reserved for counters.
  4. 4. Why does a middleware label requests by route template instead of the raw URL?

    Each distinct URL would become its own series. The template groups all /users/N requests into one series.

Summary

  • A client library keeps metrics in a registry in memory and renders them in the text exposition format; Prometheus scrapes the /metrics endpoint.
  • Declare metrics once at start-up; update them with inc, set or observe; expose the registry through a helper server or a route.
  • Use middleware for the RED signals and add hand-written metrics for business events and slow dependencies.
  • Keep label values bounded: use route templates, never user ids, request ids or raw messages.
  • Name metrics in snake_case with base units (_seconds, _bytes), end counters in _total, and initialise known label combinations at zero.

Next, see how to get metrics out of software you cannot change: Exporters: Node, Blackbox and Friends.