Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 37 ยท Operating Microservices

Distributed Tracing

Follow one request across many services: trace and span IDs, the W3C traceparent header, OpenTelemetry concepts, sampling, and a runnable mini tracer.

Metrics tell you that checkout got slow. Logs tell you what each service said. But neither answers the question you ask next: where, in the chain of six services, did the time go? A distributed trace answers it. It records the journey of one request as a tree of timed steps, across every process it touches.

Tracing builds on the correlation ID from Observability: Structured Logs and Metrics. A correlation ID tells you which log lines belong together; a trace adds structure (who called whom) and timing (how long each step took).

Anatomy of a trace

Trace The whole journey of one request.
Span One timed step inside it.
Parent link Each span knows which span called it.

A trace is a tree of spans sharing one trace ID.

1 / 2

Traces and spans

A span represents one unit of work: handling an HTTP request, a database query, a call to another service. It has a name, a start time, a duration, and attributes (key-value details such as the HTTP status or the SQL operation). A span may also carry events and an error status.

Spans are linked by parent relationships. The span for "gateway handles GET /checkout" is the parent of "orders handles POST /orders", which is the parent of "call payments", and so on. All spans of one request share a trace ID. Put together, they form a tree that a tracing tool draws as a waterfall, so you can see at a glance that the slow part is the 120 ms call to the card network.

Context propagation

The hard part of distributed tracing is context propagation: each service must tell the next one which trace it belongs to and which span is the parent. Without it, every service starts a fresh, disconnected trace.

The standard is the W3C Trace Context specification, which defines the traceparent HTTP header. It has four dash-separated fields:

TEXT
traceparent: 00-<trace-id: 32 hex>-<parent-span-id: 16 hex>-<flags: 2 hex>

The first field is the version (00), then a 128-bit trace ID, the 64-bit ID of the caller's span, and flags whose lowest bit means "this trace is sampled". An optional tracestate header carries vendor-specific extras. Older systems used their own headers (Zipkin's B3, for instance), and tracing libraries can usually read several formats.

Every hop does three things: extract the context from the incoming request, create a span that is a child of it, and inject a new context (now pointing at the new span, or at an outgoing-call span) into every outgoing request. For messages on a broker (Asynchronous Messaging and Events), the same fields go into the message headers. A service that forgets to forward the header breaks the trace in two.

Try building and parsing a header yourself. The IDs below are generated from a simple counter, not random, so the output is repeatable; real tracers use random IDs.

Node.js The traceparent header and sampling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

A valid header parses into trace ID, parent span ID and the sampled flag. A malformed or missing header returns null: a service that gets null should simply start a new trace, never crash. The last lines show sampling: the decision is derived from the trace ID, so every service that sees the same trace reaches the same decision without talking to the others.

Sampling

Recording every span of every request is expensive in CPU, network and storage, so most systems sample.

  • Head-based sampling decides at the start, usually by a fixed ratio such as 1% or 10%. It is cheap and simple, but the interesting traces (errors, slow requests) are only kept at the same rate as the boring ones.
  • Tail-based sampling collects all spans first, then keeps traces that turned out interesting (an error, or slower than a threshold). It keeps the useful ones but needs a collector that buffers whole traces.

Common practice is a low head sampling rate for normal traffic, plus rules that keep every error trace.

A mini tracer across services

The stand-in below connects four services (gateway, orders, stock, payments) in one file. Each uses extract and inject on header objects exactly as real services do with HTTP headers, and a fake clock that advances only when "work" happens so the numbers are repeatable. After the request, it prints the spans as an indented tree with start offsets and durations.

Node.js Building a trace tree across four services

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

All eight spans share one trace ID, and the indentation shows the call tree. Read the offsets: the whole request took 190 ms, and stock finished quickly (15 ms), but payments took 158 ms, of which 120 ms was the call to the card network that ended in a timeout. The tool found it by listing leaf spans (spans with no children) and sorting by duration: the slowest leaf is where to look first. This is exactly the question a tracing UI answers, only with a waterfall chart instead of text.

Notice also what the tree reveals that a flat log would not: the orders service calls stock and payments one after the other. If they do not depend on each other, calling them in parallel would cut about 20 ms, and a long chain of sequential calls is a classic latency problem that traces make visible. Latency adds up across calls, as The Fallacies of Distributed Computing warned.

OpenTelemetry

You rarely write tracers by hand. OpenTelemetry (OTel) is the vendor-neutral, open standard and set of libraries for traces, metrics and logs. Its main parts:

  • API and SDK. Your code (or an instrumentation library) creates spans through the API. The SDK decides sampling and exports the data.
  • Instrumentation. Libraries that wrap HTTP servers and clients, database drivers and message clients so that spans and header propagation happen automatically.
  • Exporters and the Collector. Services send data, commonly over the OTLP protocol, to an OpenTelemetry Collector. The collector batches, filters, samples and forwards it to a backend such as Jaeger, Tempo, Zipkin or a commercial product.
  • Resource attributes. Fields such as service.name and the deployment environment that describe who produced the data.

Because instrumentation and exporting are separated from the backend, you can switch tracing vendors by changing collector configuration, not application code.

Setting up the Node.js SDK (read-only)

This is a typical setup file for a Node.js service. It is real code, but it needs npm packages and a running collector, so the site cannot run it. Package names and options change between versions, so check the current OpenTelemetry documentation before copying it.

JavaScript
// tracing.js: load this before the rest of the app
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";

const sdk = new NodeSDK({
  serviceName: "orders",
  traceExporter: new OTLPTraceExporter({
    url: "http://otel-collector:4318/v1/traces",
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

With auto-instrumentation, incoming HTTP requests become spans and outgoing HTTP calls carry the traceparent header without further code. For business steps, add manual spans:

JavaScript
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("orders");

async function reserveStock(items) {
  return tracer.startActiveSpan("reserve stock", async (span) => {
    try {
      span.setAttribute("order.items", items.length);
      return await callStockService(items);
    } catch (error) {
      span.recordException(error);
      throw error;
    } finally {
      span.end();
    }
  });
}

Using traces well

  • Name spans by operation, not by data. Use GET /orders/{id}, not GET /orders/8841; otherwise you get unbounded variety.
  • Add useful attributes, such as the order's item count or the cache hit flag, but never secrets or personal data (Securing the Edge, Data and Secrets).
  • Link logs to traces. Put the trace ID in every log line (it can double as your correlation ID) so you can jump from a slow span to the logs of that exact request.
  • Check async boundaries. Message consumers should start a new span linked to the producer's, since the work may happen minutes later.
  • Mind the overhead. Tracing adds a little CPU and header size per call; sampling keeps the cost in check.

What is real and what is the stand-in

The traceparent format, the parent-child span model, sampling and the OpenTelemetry architecture are real. The SDK snippets are real code shown read-only. The TryIts implement the idea in plain Node with fake clocks and sequential IDs so that the output never changes; real IDs are random and real durations are measured, and the services run in separate processes.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a span represent?

    A trace is made of spans, each a timed step with a parent link and attributes.
  2. 2. What happens if a service forgets to forward the traceparent header?

    Context propagation is what links spans across processes. Without it, the next service has no parent to attach to.
  3. 3. Why is tail-based sampling attractive?

    It decides after the trace is complete, so it can keep errors and slow traces. The cost is buffering whole traces.
  4. 4. Which field of the traceparent header carries the caller's span?

    The trace ID is shared by all spans, while the parent span ID names the span that made the call.
  5. 5. What is the benefit of OpenTelemetry's collector?

    Applications export to the collector in a standard format, and the collector forwards data wherever you configure.

Summary

  • A trace is the journey of one request; spans are its timed, nested steps linked by parent IDs.
  • Context propagation (the W3C traceparent header) must be extracted and injected at every hop, including message headers.
  • Sampling controls cost; keep error and slow traces with tail-based rules where you can.
  • OpenTelemetry provides the API, instrumentation, OTLP export and the collector, independent of the backend.
  • Trace trees reveal sequential calls and slow dependencies that flat logs hide.

Next: Deployment Strategies and CI/CD.