Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 66 ยท Production Features

Observability: Metrics, Tracing and Logs

See inside a running Spring Boot app: Micrometer counters and timers, the Prometheus format, OpenTelemetry traces, log correlation, dashboards and alerts.

When something is slow at 3 a.m., you cannot attach a debugger. All you have is what the application chose to tell you while it ran. Observability is the practice of making that output good enough that you can answer new questions about the system without shipping new code. It rests on three kinds of signals: metrics (numbers over time), traces (the path of one request through many services) and logs (events with detail).

Spring Boot wires all three together through Micrometer, the metrics and tracing facade that ships with Boot. This lesson explains the vocabulary, shows the real configuration (read-only, because the real libraries and a Prometheus server cannot run on this site), and then builds two small runnable stand-ins: a mini metrics registry that prints the Prometheus text format, and a trace-id propagation demo. It builds on the Actuator lesson and on Logging.

Three signals, three questions

Metrics Is something wrong? How bad is it?
Traces Where did this one request spend its time?
Logs What exactly happened, and why?

You usually start with a metric alert, follow a trace, and finish in the logs.

1 / 2

Metrics with Micrometer

Micrometer is to metrics what SLF4J is to logging: you code against one API, and the backend (Prometheus, Datadog, CloudWatch, OTLP...) is a dependency choice. The central type is MeterRegistry. Boot creates one for you and registers a lot of meters automatically: JVM memory and garbage collection, thread counts, the HTTP server (http.server.requests), the connection pool (hikaricp.connections.*) and more.

The meter types you will use most:

  • Counter: only goes up. Orders placed, errors thrown, messages consumed. You never read the raw value; you look at its rate.
  • Gauge: a value that goes up and down and is sampled when scraped. Queue length, cache size, active sessions.
  • Timer: counts events and sums their durations, so you get both throughput and latency. Request handling, database calls.
  • DistributionSummary: like a timer, but for sizes that are not durations, such as payload bytes.

Here is how application code records its own meters. Inject the registry, build the meters once, and use them:

Java
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import org.springframework.stereotype.Service;

@Service
class CheckoutService {

    private final Counter placed;
    private final Timer paymentTimer;

    CheckoutService(MeterRegistry registry, OrderQueue queue) {
        this.placed = Counter.builder("orders.placed")
                .description("Orders accepted by the shop")
                .tag("channel", "web")
                .register(registry);
        this.paymentTimer = Timer.builder("payment.call")
                .publishPercentileHistogram()
                .register(registry);
        registry.gauge("orders.queue.size", queue, OrderQueue::size);
    }

    void checkout(Cart cart) {
        paymentTimer.record(() -> gateway.charge(cart));
        placed.increment();
    }
}

The name uses dots (orders.placed); each backend converts it to its own style. Tags (channel=web) add a dimension you can group and filter by.

The cardinality trap

Every distinct combination of tag values creates a separate time series that the monitoring system must store. A tag like channel with three values is cheap. A tag like userId or orderId with a million values can crash your metrics backend and your bill. The rule: tag values must come from a small, bounded set (status, method, route template, region). Put unbounded identifiers in logs and traces, never in metric tags. Boot follows the rule itself: http.server.requests is tagged with the route template /orders/{id}, never the real URL.

The Prometheus format

Add micrometer-registry-prometheus and expose the endpoint, and Actuator serves a plain-text page at /actuator/prometheus that Prometheus scrapes every few seconds:

PROPERTIES
management.endpoints.web.exposure.include=health,prometheus
management.metrics.tags.application=${spring.application.name}

The page looks like this (shortened):

TEXT
# HELP orders_placed_total Orders accepted by the shop
# TYPE orders_placed_total counter
orders_placed_total{application="shop",channel="web"} 42.0
# TYPE payment_call_seconds summary
payment_call_seconds_count{application="shop"} 42
payment_call_seconds_sum{application="shop"} 12.6
payment_call_seconds_max{application="shop"} 0.9

Notice the conversions: dots became underscores, the counter gained a _total suffix and a timer became _seconds_count / _seconds_sum / _seconds_max. Average latency is sum / count over a time window, which is why timers record both. The common-tags line adds application="shop" to every series so that you can tell services apart on a shared dashboard.

A mini registry you can run

The real MeterRegistry is a library we cannot load here, so this stand-in keeps the same ideas in about sixty lines: meters identified by name plus tags, counters, timers recorded from fixed durations (so the output is deterministic), and a text exposition in the Prometheus style. What is real: the naming conversions and the structure of the output. What is the stand-in: the registry itself, with no gauges, histograms or scraping.

Java A mini Micrometer registry

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Look at the output: the two orders_placed_total lines are different series because their tags differ, and the average latency comes from dividing the sum by the count, which is exactly how a Prometheus query computes it over a time window.

Percentiles and alerts

An average hides pain: if 99 requests take 50 ms and one takes 5 s, the average looks fine but one user in a hundred is suffering. That is why timers can publish a histogram (publishPercentileHistogram(), or management.metrics.distribution.percentiles-histogram.http.server.requests=true). The monitoring system then computes percentiles such as p95 and p99 from the buckets, and can combine buckets from many instances, which is something percentiles calculated inside each JVM cannot do.

Good alerts follow the RED method for services:

  • Rate: requests per second (sudden drop = outage upstream; sudden rise = traffic spike or retry storm).
  • Errors: share of responses with status 5xx or of calls that threw.
  • Duration: p95 or p99 latency against a target.

Alert on symptoms users feel (errors and slowness), not on every internal cause such as "CPU above 70%". Keep a handful of alerts that always mean "wake someone up", and put everything else on dashboards.

Tracing across services

A metric tells you that checkout is slow. A trace tells you where. When a request enters your system, tracing creates a trace id; each unit of work along the way (an incoming HTTP call, a database query, an outgoing call) becomes a span with its own span id, start time and duration, pointing at its parent span. The trace id travels to downstream services in an HTTP header (the W3C traceparent header), so spans from many services join into one tree.

sequenceDiagram
    participant U as Browser
    participant S as shop service
    participant P as payment service
    U->>S: GET /checkout
    Note over S: trace abc123, span A
    S->>P: POST /charge (traceparent: abc123-A)
    Note over P: same trace abc123, span B (parent A)
    P-->>S: 200 OK
    S-->>U: 200 OK

The trace id stays the same across services; each hop adds a new span whose parent is the caller's span.

In Boot you add the Micrometer Tracing bridge for OpenTelemetry and an exporter, then point it at a collector or tracing backend (Jaeger, Tempo, Zipkin or a vendor):

XML
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
PROPERTIES
management.tracing.sampling.probability=0.1
management.otlp.tracing.endpoint=http://collector:4318/v1/traces

Boot's auto-configuration then creates spans for incoming requests and for calls made through RestClient, RestTemplate and WebClient when they are built from the Boot-provided builders (a client created with new RestTemplate() skips the instrumentation, a common reason for "my trace ends at the first hop"). The default sampling probability is 0.1: only one request in ten is recorded. Use 1.0 while developing and a low value in busy production systems, because tracing every request costs CPU, network and storage.

For your own business steps, use the Observation API, which records a timer metric and a span from one place:

Java
@Observed(name = "checkout.price-calculation")
BigDecimal calculatePrice(Cart cart) { ... }

@Observed needs the ObservedAspect bean and AOP on the classpath. You can also use an ObservationRegistry directly when you need to add tags or error handling.

Correlation: connecting logs to traces

Metrics and traces are only half the story; the detail ("which customer, which SQL, which exception") lives in logs. The glue is the trace id in every log line. When tracing is active, Boot places traceId and spanId into the MDC (the per-thread logging context from the Logging lesson), and the default log pattern prints them. A log search for the trace id from a slow trace then returns every log line, from every service, that belongs to that one request.

TEXT
2026-11-08T10:15:30.120Z  INFO 4711 --- [shop] [nio-8080-exec-3] [abc123def4567890abc123def4567890-1a2b3c4d5e6f7a8b] c.e.s.CheckoutService : Charging 42.00 EUR

Three rules keep this useful. Log structured output (JSON) in production so the id is a searchable field. Do not log secrets or personal data, since logs are copied to many systems. And remember that MDC is thread-bound: when you hand work to another thread (an @Async method, an executor), the context does not follow automatically unless Boot's instrumented executors or a context-propagation setup carry it over. The next TryIt shows why.

A trace-id demo

This stand-in generates trace and span ids from a counter (so the output is the same every run), propagates them across two "services" in a header map, and writes log lines that include them. It also shows the thread-bound context being lost on a raw new thread. What is real: the parent/child structure and header propagation. What is the stand-in: ids are not random, the header is not the real traceparent format, and ThreadLocal plays the part of the MDC.

Java Propagating a trace id

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Both shop and payment lines carry trace-001, while the payment span's parent is the shop's span. The last line shows [no trace]: that is the reason instrumented executors exist in real Boot.

Putting it together on a dashboard

A practical starting dashboard for any Boot service has one row per signal: request rate, error rate and p95 latency from http.server.requests; JVM heap and garbage-collection pause time; connection-pool usage (hikaricp.connections.active against the maximum); and two or three business counters (orders placed, payments failed). When a panel turns red, you click through to traces for the same time window, then to the logs of one trace id. Plan that path before the incident, not during it.

Two operational notes: keep /actuator/prometheus off the public internet (see Actuator, and expose it on a separate management port if you can), and measure the cost of observability itself, since tracing every request and recording high-cardinality tags are the usual ways it hurts performance.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is tagging a meter with userId a bad idea?

    Cardinality is the number of distinct tag-value combinations. Keep tag values to a small bounded set and put identifiers in logs or traces instead.
  2. 2. What does a Micrometer Timer give you that a Counter does not?

    A timer records count, total time and maximum, so average latency is sum divided by count, and with a histogram you can compute percentiles.
  3. 3. A trace shows a request ending at the first service although it called a second one. What is a likely cause?

    Boot instruments clients built from its builders. A hand-made client sends no trace context, so the downstream spans start a new trace.
  4. 4. What do traceId and spanId in the MDC enable?

    Putting the trace id in every log line correlates logs with traces, so you can jump from a slow trace to its detailed log messages.
  5. 5. Which set of signals does the RED method watch?

    Rate, Errors and Duration describe what users experience of a service, which makes them the best basis for alerts.

Summary

  • Observability combines metrics (numbers), traces (one request's path) and logs (detail); Micrometer is Boot's facade for the first two.
  • Use Counter, Gauge, Timer and DistributionSummary; keep tag values bounded to avoid cardinality explosions.
  • micrometer-registry-prometheus plus Actuator exposes /actuator/prometheus; names become snake_case, counters get _total, timers become _seconds_count/sum/max.
  • Tracing needs the Micrometer Tracing bridge and an exporter; the default sampling is 10%, and clients must come from Boot's builders to propagate context.
  • Put the trace id into every log line, alert with the RED method, and secure the metrics endpoint.

Next: Scheduling and Asynchronous Work.