Skip to content
Tutorials Center
My learning

Topics

Lesson 108 of 119 ยท Spring Cloud

Distributed Tracing with Micrometer and Zipkin

Follow one request across Java microservices. Learn spans, W3C traceparent propagation, Micrometer Tracing with OpenTelemetry or Brave, Zipkin, sampling and log correlation.

A user clicks "Place order" and waits three seconds. The request went through a gateway, the order service, the inventory service, a payment provider and two databases. Which one was slow? Each service's metrics say "a bit slower than usual", and each service's logs contain thousands of unrelated lines. Distributed tracing answers the question directly: it records the path of one request through every service, with timings, and draws it as a timeline.

Actuator, Metrics and Observability introduced traces, spans and the traceparent header, and passed a trace id between two services. This lesson builds the complete picture: the exact header format and its rules, client and server spans, a collector that assembles spans from three services into a timeline tree, sampling that keeps traces complete, and correlating logs with trace ids even across thread pools. Then you'll see how Micrometer Tracing, OpenTelemetry and Zipkin do all of it in a Spring Boot application.

As in the rest of this chapter, the runnable examples rebuild each mechanism with the JDK (and Jackson), because Micrometer, OpenTelemetry and Zipkin aren't available in our code runner. The real configuration is shown read-only.

Traces, spans and their relationships

A trace is the whole story of one request. It is a tree of spans, and each span describes one timed operation:

  • trace id: 16 random bytes shared by every span in the trace.
  • span id: 8 random bytes identifying this span.
  • parent id: the span id of the operation that caused this one (empty for the root span).
  • name: a low-cardinality description such as GET /orders/{id} or SELECT orders, never one with ids in it.
  • kind: SERVER (handling an incoming request), CLIENT (making an outgoing call), PRODUCER/CONSUMER (messaging), or internal.
  • start time and duration, plus tags (also called attributes: http.response.status_code=200, db.system=postgresql) and timestamped events.

One HTTP call between two services usually produces two spans: a CLIENT span in the caller and a SERVER span in the callee, whose parent is that client span. The gap between them is network time and queuing, which is often exactly what you are hunting for.

Propagating context: the traceparent header

Spans only join into one trace if the caller tells the callee which trace it belongs to. The W3C Trace Context standard defines the traceparent header for this:

TEXT
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
             |  |                                |                |
             |  trace id (32 lowercase hex)      parent id (16)   flags (01 = sampled)
             version (00)

The parent id is the span id of the caller's client span. The flags byte's lowest bit says whether the caller is recording this trace (sampled). A second header, tracestate, carries vendor-specific data and must be passed along unchanged. The rules matter, because a receiver that accepts a malformed header can corrupt traces: an all-zero trace id or parent id is invalid, uppercase hex is invalid, version ff is invalid, and a version-00 header has exactly 55 characters.

Before W3C Trace Context, Zipkin's B3 headers were the standard (X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId, X-B3-Sampled, or one combined b3 header). You still meet them in older systems, and libraries can read and write both.

Java Parsing and validating traceparent headers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The outgoing header keeps the trace id and the sampled flag and replaces only the parent id. When a service receives an invalid header, the specification says to ignore it and start a new trace, which is exactly what tracing libraries do.

Spans from three services, assembled by a collector

Each service sends its finished spans to a collector. Zipkin is the classic open-source one: it accepts spans over HTTP at POST /api/v2/spans as a JSON array, stores them, and its web UI (port 9411) draws traces. Jaeger and Grafana Tempo are popular alternatives, and the OpenTelemetry Collector often sits in between, receiving spans in the OTLP format and forwarding them to any backend.

The next program runs the whole pipeline on 127.0.0.1: a gateway, an order service and an inventory service, each with its own HTTP server, plus a Zipkin-like collector. Each service continues the trace from the incoming traceparent, creates CLIENT spans for outgoing calls and local spans for database work, and reports every finished span in Zipkin's v2 JSON format (times in microseconds). At the end, the program builds the tree and prints a timeline.

Java Three services, a span collector and a timeline tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Your durations differ slightly on each run, but the shape is always the same. The gateway's server span is the root and covers the whole request. Under it, the gateway's CLIENT span covers the call to orders, and the orders SERVER span sits inside it. Orders first calls inventory (another client/server pair, with the SELECT stock span inside), then inserts the order. Reading such a timeline, you immediately see which operation took the time, which calls ran one after another and could run in parallel, and how much time vanished between a client span and its server span.

Like real tracers, the demo's end() only puts the span into a queue, and a background reporter thread sends spans in batches, so tracing never slows a request down or fails it when the collector is unreachable. Production reporters also cap the queue size and drop spans rather than run out of memory.

Sampling: keeping costs under control

Recording every span of every request is expensive at scale: storage, network and CPU. Sampling keeps a fraction. The crucial rule is that the decision is made once, at the root, and propagated in the sampled flag. If each service flipped its own coin, most stored traces would be missing pieces. The program compares three strategies on 10,000 requests that each pass through three services:

Java Head-based sampling that keeps traces complete

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Independent decisions keep almost no complete traces and thousands of useless fragments. Propagating the root's decision (what OpenTelemetry calls a parent-based sampler) and deciding from the trace id itself both keep about 10% of requests, every one of them complete. Real systems combine the two: the root uses a trace-id ratio, and everyone downstream respects the incoming flag.

This is head-based sampling: the decision happens before anyone knows whether the request will be slow or fail. Tail-based sampling, done in the OpenTelemetry Collector or the tracing backend, buffers all spans of a trace for a few seconds and then keeps the interesting ones: all errors, all slow requests, and a small share of the rest. It costs more infrastructure but answers the questions you actually ask during incidents.

Correlating logs with traces

A trace tells you where time went; logs tell you what happened there. The connection is the trace id in every log line. Tracing libraries put traceId and spanId into the logging MDC (the thread-local context described in Logging with SLF4J and Logback), and the log pattern prints them. From a slow span in Zipkin you search the logs for its trace id and get every line from every service for that request.

The weak spot is threads. The MDC belongs to a thread, so work handed to an executor loses it, and a pooled thread that isn't cleaned up carries a stale trace id into the next task. This program shows both bugs and the fix: capture the context when the task is submitted and restore it inside the task.

Java Trace ids in logs, across thread pools

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The plain submit logs empty ids, so the e-mail step disappears from the request's story. The wrapped task logs the right ids. The last line is the nastier bug: an unrelated job logged a trace id that belongs to someone else's request, which sends whoever investigates on a wrong path. In Spring applications, the Micrometer Context Propagation library does the capture-and-restore for you: Spring Boot applies it to @Async executors and scheduled tasks when configured, and Reactor pipelines get it with spring.reactor.context-propagation=auto. Java's ScopedValue (see Virtual Threads) avoids stale values by design, because a binding ends with its scope.

Micrometer Tracing in Spring Boot

Spring Boot instruments your application through the Micrometer Observation API. One Observation around an operation feeds several handlers: one records a timer metric, one creates a span, others can log. That's why a single instrumentation point gives you both the http.server.requests metric and a span. The last example shows the idea.

Java One observation, two signals

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Three observations produced three spans and two timer series, one per country tag value. Tags marked low-cardinality go to both metrics and spans; high-cardinality values such as an order id belong only on spans, because they would explode the metrics, as explained in the Actuator lesson.

Choosing the libraries

Micrometer Tracing is a facade with two bridges, like SLF4J with two logging backends:

Bridge Tracer underneath Typical exporter
micrometer-tracing-bridge-otel OpenTelemetry SDK OTLP to an OpenTelemetry Collector, Tempo, Jaeger; or Zipkin
micrometer-tracing-bridge-brave Brave (Zipkin's Java tracer) zipkin-reporter-brave to Zipkin

OpenTelemetry is the industry direction: one vendor-neutral standard for traces, metrics and logs, supported by every major backend. Choose Brave mainly when you already run a Zipkin-based setup. Spring Boot 4 also offers spring-boot-starter-opentelemetry, which brings the OTel bridge and OTLP export in one dependency.

A Zipkin setup with Spring Boot 3 looks like this (Boot 4 reorganized some tracing property names, so check the reference for your version):

XML
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<dependency>
  <groupId>io.zipkin.reporter2</groupId>
  <artifactId>zipkin-reporter-brave</artifactId>
</dependency>
PROPERTIES
spring.application.name=orders
management.tracing.sampling.probability=1.0
management.zipkin.tracing.endpoint=http://localhost:9411/api/v2/spans
management.tracing.propagation.type=w3c
logging.pattern.correlation=[${spring.application.name:},%X{traceId:-},%X{spanId:-}] 
Terminal
docker run -d -p 9411:9411 openzipkin/zipkin    # then open http://localhost:9411

The sampling probability defaults to 0.1; 1.0 is for development. The correlation pattern makes every log line carry the trace and span id, and with structured JSON logging the ids become searchable fields.

Instrumentation rules that save hours

  • Build HTTP clients from the injected builders (RestClient.Builder, WebClient.Builder, RestTemplateBuilder). A RestClient.create() made by hand has no observation support, so the traceparent header is never sent and the trace breaks at that call.

  • Add your own spans only for meaningful work, either with @Observed(name = "tax.calculation") (needs AOP support) or programmatically:

    Java
    Observation.createNotStarted("tax.calculation", observationRegistry)
        .lowCardinalityKeyValue("country", country)
        .highCardinalityKeyValue("order.id", orderId)
        .observe(() -> taxService.calculate(order));
    
  • Messaging propagates context in message headers; Spring Kafka and Spring AMQP do it when observation is enabled on the template and listener container (see Messaging with Kafka and RabbitMQ).

  • Baggage carries business context (a tenant id, a user tier) through the whole trace: management.tracing.baggage.remote-fields=tenant-id and, to see it in logs, management.tracing.baggage.correlation.fields=tenant-id. Never put secrets or personal data in baggage; it travels in plain headers to every service.

  • Agree on one propagation format across all services and the gateway, or traces split at the boundary.

  • The OpenTelemetry Java agent (-javaagent:opentelemetry-javaagent.jar) is the zero-code alternative: it instruments common libraries at startup. Use either the agent or Micrometer's instrumentation in a service, not both, or you get duplicate spans.

Tracing completes the resilience story from Circuit Breakers and Resilience: retries, fallbacks and open breakers show up as spans and tags, so you can see exactly which attempt failed and how long the user waited.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Service A receives traceparent 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01 and calls B from a client span with id 53995c3f42cd8ad8. What does A send to B?

    The trace id and flags stay the same; the parent-id field is replaced by the span id of A's outgoing client span.
  2. 2. Each of three services samples 10% of requests independently. What do you get in the tracing backend?

    A trace is complete only if all three record it, which happens for about 0.1% of requests. The root's decision must be propagated in the sampled flag.
  3. 3. Log lines written inside a CompletableFuture.runAsync task show an empty traceId, while lines on the request thread show it. Why?

    MDC values belong to a thread. Executors must capture the context on submit and restore it in the task, which context-propagation libraries do for you.
  4. 4. A service builds its client with RestClient.create() instead of the injected RestClient.Builder. What happens to traces?

    Instrumentation is attached to the builders Spring Boot configures. A hand-made client has no observation support and doesn't propagate context.
  5. 5. You want to keep every trace that contains an error, but only 5% of successful ones. Which approach can do that?

    Head-based samplers decide before the outcome is known. Only tail-based sampling can choose traces by their errors or latency.

Summary

  • A trace is a tree of spans sharing a 16-byte trace id; each span has an 8-byte id, a parent, a kind (SERVER, CLIENT, PRODUCER, CONSUMER, internal), timings and tags. One remote call usually yields a client span and a server span.
  • Context travels in the W3C traceparent header (version-traceid-parentid-flags); the caller replaces the parent id with its client span id. Invalid headers are ignored and a new trace starts. B3 headers are the older Zipkin format.
  • Services report spans asynchronously to a collector such as Zipkin, Jaeger, Tempo or the OpenTelemetry Collector, which assembles them into timelines.
  • Sample at the head and propagate the decision so traces stay complete; use tail-based sampling to keep all errors and slow requests.
  • Put trace ids in every log line through the MDC, and propagate context to thread pools and reactive code.
  • In Spring Boot, the Micrometer Observation API produces metrics and spans; choose the OpenTelemetry or Brave bridge, set the sampling probability and exporter, and always build clients from the injected builders.

Next: Project: Library Database with JDBC