Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 36 ยท Observability

Distributed Tracing in Istio: Spans, Headers and Sampling

Understand how Istio produces trace spans, why applications must forward trace headers, how to set sampling with the Telemetry API and how to send traces to Zipkin, Jaeger or OpenTelemetry.

Metrics tell you that something is slow. Traces tell you where. A trace follows one user request across every service it touches, as a tree of timed spans. Istio's proxies generate spans automatically for each hop, so you get a service-level picture of a request without installing a tracing library. There is one catch, and it is the most common reason tracing "does not work" in a mesh: the application itself must forward a few headers.

Everything here is conceptual and read-only. The product cannot run on this site, so the runnable parts are Python stand-ins that model spans and sampling.

What the proxy does, and what it cannot do

When a request enters a sidecar, Envoy starts a span (or continues the one named in the incoming headers). It records start time, duration, upstream service, status code and the proxy's own timing, then exports the span to a collector. Envoy sees only a single connection at a time. When service A receives a request and then calls service B, the proxy has no way to know that the outgoing call belongs to the incoming one, unless the application copies the trace headers from the inbound request to the outbound request.

sequenceDiagram
  participant U as Client
  participant GW as Ingress gateway
  participant A as frontend (app + proxy)
  participant B as reviews (app + proxy)
  U->>GW: GET /product
  GW->>A: trace id T, span 1
  A->>B: needs the SAME trace id T (app copies headers)
  B-->>A: 200
  A-->>GW: 200
  GW-->>U: 200

If the frontend does not copy the headers, the call to reviews starts a brand new trace and you see two unrelated traces.

Headers to propagate

Istio's proxies understand several formats. Applications should forward whichever they receive:

  • W3C Trace Context: traceparent, tracestate (the modern standard, used by OpenTelemetry).
  • B3 (Zipkin): x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags, or the single header b3.
  • Envoy's own: x-request-id (useful for correlating logs) and x-envoy-attempt-count.

Most applications get this for free by using an OpenTelemetry SDK or auto-instrumentation, which propagates context through HTTP clients automatically.

Forward the headers you receive; do not invent ids. The proxy creates the root span if the client sent nothing.

Configuring a tracing backend

Providers are declared once in the mesh configuration, then selected with a Telemetry resource.

YAML
# mesh config (istiod ConfigMap or IstioOperator values)
meshConfig:
  extensionProviders:
    - name: otel-tracing
      opentelemetry:
        service: otel-collector.observability.svc.cluster.local
        port: 4317
    - name: zipkin
      zipkin:
        service: zipkin.istio-system.svc.cluster.local
        port: 9411
YAML
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-tracing
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: otel-tracing
      randomSamplingPercentage: 1.0     # one request in a hundred
      customTags:
        environment:
          literal:
            value: production

The Jaeger and Zipkin demo add-ons (samples/addons/jaeger.yaml) are for learning only. In production, send spans to an OpenTelemetry Collector, which can batch, filter and forward them to whatever backend you use.

Sampling: how much is enough?

Tracing every request is expensive in CPU, network and storage. The usual approach is head-based sampling: the first proxy decides, with a fixed probability, whether this trace is recorded, and every other hop follows the decision carried in the headers (x-b3-sampled or the sampled flag of traceparent). At 1 percent you keep storage small but need enough traffic to catch rare problems.

Try it: build a trace tree from spans

Each span knows its parent. This stand-in rebuilds the tree and prints where time went, which is exactly what a trace viewer does.

Python Render a trace as a tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The span with the largest self time (its duration minus its children) is usually where to look first.

Try it: header propagation and sampling

This second stand-in shows why a missing header breaks a trace, and how a sampling rate affects the chance of catching a rare slow request.

Python Header propagation check and sampling odds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a trace break between two services even though both have sidecars?

    The proxy cannot link an outbound call to an inbound one on its own; the application must propagate the headers.
  2. 2. What does randomSamplingPercentage: 1.0 mean in a Telemetry resource?

    The value is a percentage, so 1.0 samples about one request in a hundred.
  3. 3. Which header format is the W3C standard?

    traceparent and tracestate are W3C Trace Context headers; B3 is the older Zipkin format.
  4. 4. Where is the sampling decision made in head-based sampling?

    The first proxy decides and the decision travels in the headers so a trace is either complete or absent.

Summary

  • Proxies create a span per hop automatically, but applications must forward trace headers.
  • Use W3C traceparent (OpenTelemetry) or B3 headers; forward x-request-id too.
  • Declare providers in meshConfig.extensionProviders and select them with Telemetry.
  • Sample a small percentage in production; the decision is made at the first hop.
  • Self time per span points at the slow service.

Next: Access logs and the Telemetry API