Skip to content
Tutorials Center
My learning

Topics

Lesson 81 of 119 ยท Spring Boot

Actuator, Metrics and Observability

Make Java services observable - Spring Boot Actuator health and info endpoints, Micrometer metrics, Prometheus, distributed tracing and structured production logs.

Once your service runs in production, you can't attach a debugger or read System.out. You need the service to tell you, continuously, whether it is healthy, how fast it is, how often it fails and where time goes when a request crosses five services. That ability to understand a running system from the outside is called observability.

Observability rests on three kinds of signals. Metrics are numbers over time (requests per second, error rate, latency percentiles, heap usage) that are cheap to store and great for dashboards and alerts. Traces follow one request through every service it touches. Logs record individual events in detail. Health checks sit next to them and answer one blunt question: should this instance receive traffic?

In Spring Boot, Actuator provides health and management endpoints, Micrometer provides metrics and tracing, and the logging system adds structured output. In this lesson you will build small versions of each piece with plain Java, so you know what the numbers mean, and then see the Spring Boot configuration that gives you the real thing. Logging basics are covered in Logging with SLF4J and Logback, and the JVM tools in Debugging and Profiling Tools.

Health checks

A load balancer or Kubernetes calls a health endpoint every few seconds. If it answers 200 OK, the instance gets traffic; if it answers 503 Service Unavailable, traffic goes elsewhere. A useful health check combines components: is the database reachable, is the disk nearly full, is the message broker connected?

The rules for aggregating are simple: the overall status is UP only if every component is UP. Here is that idea on the JDK HttpServer, with a real H2 database check (Connection.isValid) and a disk-space check, returning JSON shaped like Actuator's:

Java A health endpoint with component checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Liveness versus readiness

Kubernetes distinguishes two questions, and mixing them up causes outages:

  • Liveness: "Is this process broken beyond repair?" If liveness fails, Kubernetes restarts the container. Only fail it for unrecoverable states (a deadlock, a corrupted internal state). Never include the database in liveness: when the database has a hiccup, every instance would restart at once, making things worse.
  • Readiness: "Can this instance serve requests right now?" If readiness fails, the instance is removed from the load balancer but keeps running. Failing readiness during startup, during a graceful shutdown, or while a critical dependency is down is fine.

Spring Boot exposes both as health groups at /actuator/health/liveness and /actuator/health/readiness. They are enabled automatically when the app detects Kubernetes, or with management.endpoint.health.probes.enabled=true. Deploying Java Applications shows the matching Kubernetes probe configuration.

Spring Boot Actuator

Add one starter:

XML
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

Actuator creates many endpoints under /actuator, but only health is exposed over HTTP by default, because the others reveal internals. You opt in to what you need:

YAML
management:
  endpoints:
    web:
      exposure:
        include: health, info, metrics, prometheus
  endpoint:
    health:
      show-details: when-authorized   # never, always or when-authorized
      probes:
        enabled: true
  info:
    env:
      enabled: true
    java:
      enabled: true
  server:
    port: 9090                       # optional: management endpoints on a separate port

info:
  app:
    name: library-service
    team: platform

The most useful endpoints:

Endpoint What it shows
health Aggregated status plus components (db, diskSpace, redis, ... added automatically for what's on the classpath)
info Free-form app info: info.* properties, build info, Git commit, Java version
metrics Lists meter names; /actuator/metrics/{name} shows values and available tags
prometheus All metrics in Prometheus text format (needs micrometer-registry-prometheus)
loggers Read and change log levels at runtime with a POST, without a restart
env, configprops Resolved configuration (sensitive values are masked)
threaddump, heapdump Diagnostics, as in Debugging and Profiling Tools
mappings, beans, scheduledtasks What Spring wired up: URL mappings, beans, @Scheduled jobs

Treat env, configprops, heapdump and loggers as admin tools. A heap dump contains every secret in memory. Expose them only on a separate management port that is not reachable from the internet, or protect them with Spring Security, as shown in Spring Security.

A custom health indicator

Any bean implementing HealthIndicator joins the health endpoint. The bean name minus the HealthIndicator suffix becomes the component name, so this one appears as catalogApi:

Java
@Component
class CatalogApiHealthIndicator implements HealthIndicator {

    private final CatalogClient catalog;

    CatalogApiHealthIndicator(CatalogClient catalog) {
        this.catalog = catalog;
    }

    @Override
    public Health health() {
        try {
            Duration latency = catalog.ping();
            return Health.up().withDetail("latencyMs", latency.toMillis()).build();
        } catch (Exception e) {
            return Health.down(e).build();
        }
    }
}

Keep health checks fast and cheap: they run every few seconds on every instance. Use a short timeout, and think twice before letting an external dependency take your whole instance out of readiness. Boot 4 moved the health classes to new packages as part of its modularization; your IDE will import the right ones.

Metrics with Micrometer

Micrometer is to metrics what SLF4J is to logging: one API, many backends (Prometheus, Datadog, New Relic, CloudWatch, OTLP and more). Its core types are:

  • Counter: a value that only goes up, such as orders placed or errors. Dashboards show its rate.
  • Gauge: a current value that goes up and down, such as queue size or active connections. Micrometer samples a function when it is scraped.
  • Timer: counts events and records their durations, giving count, total time, max and optional percentiles or histograms.
  • Distribution summary: like a timer for non-time values, such as response sizes in bytes.

Every meter has a name and optional tags (dimensions). http.server.requests with tags method=GET, uri=/books/{id}, status=200 is one time series; a different status is another. This makes tags powerful, and dangerous: each distinct combination is a separate series stored in your monitoring system.

Never use unbounded values as tags: user ids, email addresses, full URLs with ids (/books/1234), or exception messages. That creates a cardinality explosion: millions of series that slow down or crash your metrics backend. Use templates like /books/{id} and a small set of outcome values.

The next example implements a tiny registry with tagged counters and timers, records simulated traffic, and prints it in the Prometheus exposition format, the plain-text format that Prometheus scrapes from /actuator/prometheus:

Java Counters, timers, tags and the Prometheus format

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

A few things to notice. Prometheus names use underscores and unit suffixes (_seconds, _total); Micrometer converts its dotted names automatically. A timer exports count and sum, so average latency is sum / count, and over time Prometheus computes rates from them. Averages hide slow outliers, which is why the max is exported and why production setups add histograms to compute percentiles such as p95 and p99.

Metrics in Spring Boot

With Actuator on the classpath, Spring Boot registers a MeterRegistry and many meters automatically: http.server.requests for every MVC endpoint (with uri as a template, safe for cardinality), jvm.memory.used, jvm.gc.pause, jvm.threads.live, process.cpu.usage, hikaricp.connections.active, logback.events, cache statistics, and more. Add micrometer-registry-prometheus to get /actuator/prometheus.

Your own business metrics are a few lines:

Java
@Service
class LoanService {

    private final Counter loansCreated;
    private final Timer checkoutTimer;

    LoanService(MeterRegistry registry, LoanRepository loans) {
        this.loansCreated = Counter.builder("library.loans.created")
            .description("Books lent to members")
            .tag("channel", "web")
            .register(registry);
        this.checkoutTimer = registry.timer("library.checkout.duration");
        Gauge.builder("library.loans.overdue", loans, LoanRepository::countOverdue).register(registry);
        // ...
    }

    Loan checkout(long bookId, long memberId) {
        return checkoutTimer.record(() -> {
            Loan loan = doCheckout(bookId, memberId);
            loansCreated.increment();
            return loan;
        });
    }
}

Percentile histograms are enabled per meter in configuration, for example management.metrics.distribution.percentiles-histogram.http.server.requests=true. Prometheus then computes p99 latency across all instances with histogram_quantile, and Grafana draws it.

JVM metrics come from MXBeans

The JVM metrics Micrometer publishes are read from the platform MXBeans in java.lang.management. You can read the same values yourself:

Java JVM metrics from MXBeans

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The exact numbers depend on the machine and the garbage collector the JVM chose. Our code runner starts the JVM with one CPU and the Serial collector, whose two halves report as Copy and MarkSweepCompact; on typical servers you see G1 instead. Docker for Java Developers explains why that matters in containers. If jvm_memory_used_bytes for the heap keeps rising after every garbage collection, you probably have a memory leak.

Distributed tracing

A request to an API gateway may call a loan service, which calls the catalog service and the member service. When it takes three seconds, which hop was slow? Metrics show that latency is high; a trace shows where.

A trace is a tree of spans. Each span is one unit of work (an incoming HTTP request, a database query, an outgoing call) with a start time, duration, name and tags. All spans of one request share a trace id; each span has its own span id and knows its parent. When a service calls another, it passes the trace context in a header. The standard is W3C Trace Context:

TEXT
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             version-trace id (16 bytes)---------span id (8 bytes)-flags (01 = sampled)

Each service creates a child span, sends spans to a collector (Zipkin, Jaeger, Grafana Tempo, or an OpenTelemetry Collector), and writes the trace id into its logs. Then you can jump from a slow trace to the exact log lines of that request in every service.

This example runs two "services" in one program. The front service starts a trace, calls the back service with a traceparent header, and both log with the trace id. The current span is held in a ScopedValue (final since Java 25), a modern alternative to ThreadLocal for passing context down a call chain:

Java Propagating a trace id between two services

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The ids are random on each run, but the pattern is always the same: both services log the same trace id, and the members service's parent span id equals the span id that loans sent. Tracing systems assemble the tree from exactly these relationships.

Tracing in Spring Boot

Spring Boot instruments incoming and outgoing HTTP calls (RestClient, WebClient), JDBC (with an extra library), messaging and @Observed methods through the Micrometer Observation API. One observation produces both a timer metric and a span. With Spring Boot 3, add the OpenTelemetry bridge and an exporter:

XML
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Boot 4 also offers a single spring-boot-starter-opentelemetry. Then configure where spans go and how many to keep:

PROPERTIES
management.tracing.sampling.probability=0.1
management.otlp.tracing.endpoint=http://otel-collector:4318/v1/traces

The default sampling probability is 10%, which keeps tracing cheap for high-traffic services; set 1.0 in development to see every request. Spring Boot also puts traceId and spanId into the logging MDC automatically, so every log line written during a request carries its trace id.

An alternative is the OpenTelemetry Java agent: a JAR passed with -javaagent: that instruments common libraries without code changes. Either way, use the same propagation format (W3C traceparent) across all services, or traces break at service boundaries.

Logging in production

Logs complete the picture, but in production nobody reads log files on servers. Logs are shipped to a central system (Elasticsearch/OpenSearch, Loki, Splunk, a cloud logging service) and searched. That changes how you should log:

  • Write to standard output. In containers, the platform collects stdout; don't manage log files yourself.
  • Use structured (JSON) logs so fields like level, logger, traceId and orderId are searchable without regular expressions. Since Spring Boot 3.4, one property does it: logging.structured.format.console=ecs (also logstash or gelf).
  • Include correlation ids: the trace id (automatic with tracing) and business keys via MDC.
  • Pick levels deliberately: ERROR for things a human must look at, WARN for degraded but handled situations, INFO for important business events, DEBUG off in production (turn it on temporarily for one logger through the loggers endpoint).
  • Never log secrets or personal data: passwords, tokens, full card numbers, and be careful with emails and addresses.

A structured log line looks like this (shortened):

JSON
{"@timestamp":"2026-10-17T09:30:12.481Z","log.level":"INFO","message":"Loan created","service.name":"library-service","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","spanId":"00f067aa0ba902b7","loanId":981}

Alerts and the golden signals

Collecting data is only useful if someone acts on it. Most teams start with the four golden signals for each service: latency (p95/p99, separately for successful and failed requests), traffic (requests per second), errors (rate of 5xx responses and failed jobs), and saturation (CPU, memory, connection pool usage, queue length). Alert on symptoms users feel, such as error ratio or latency above an agreed objective for several minutes, rather than on every CPU spike. Each alert should link to a dashboard and, ideally, a short runbook.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Your readiness check includes the database. The database is unavailable for 20 seconds. What happens in Kubernetes?

    A failed readiness probe removes the pod from service endpoints but keeps it running. A failed liveness probe would restart it, which is why liveness should never depend on the database.
  2. 2. A developer adds a userId tag to the http.server.requests timer. What is the main risk?

    Each distinct tag combination is a separate series. Unbounded values such as user ids, raw URLs or messages can create millions of series.
  3. 3. You add spring-boot-starter-actuator and request /actuator/env. What do you get by default?

    Only health is exposed over HTTP by default. Endpoints like env must be added to management.endpoints.web.exposure.include, and should be protected.
  4. 4. A timer reports _seconds_count 200 and _seconds_sum 30 for one minute. What do you know?

    Sum divided by count gives the average. Percentiles need histograms, and the slowest request is the max value.
  5. 5. Service A calls service B. In the tracing system, B's spans appear as a separate trace. What is the most likely cause?

    Traces join only through propagated context. If A doesn't send traceparent, or B expects another format, B starts a new trace.

Summary

  • Observability = metrics (numbers over time), traces (one request across services) and logs (detailed events), plus health checks for traffic routing.
  • Health aggregates components: UP only if all are UP; 503 otherwise. Liveness restarts, readiness removes from the load balancer; keep dependencies out of liveness.
  • Actuator exposes only health over HTTP by default; include others explicitly and protect sensitive ones (env, heapdump, loggers), ideally on a separate management port.
  • Micrometer offers counters, gauges, timers and distribution summaries with tags; avoid high-cardinality tags. Prometheus scrapes /actuator/prometheus; use histograms for percentiles.
  • Traces are trees of spans sharing a trace id, propagated with the W3C traceparent header; Spring Boot uses the Micrometer Observation API with an OpenTelemetry bridge and puts trace ids into logs.
  • In production, log JSON to stdout with trace ids, choose levels deliberately and never log secrets.

Next: Scheduling, Caching and Async Tasks covers background jobs, caches and asynchronous methods, and the metrics you just learned will tell you whether they work.