Skip to content
Tutorials Center
My learning

Topics

Lesson 61 of 78 ยท Operations and Observability

Tracing and Service Meshes

Follow a request across microservices with distributed tracing and OpenTelemetry, and learn what Istio and Linkerd add: mTLS, retries, sidecarless meshes.

Metrics tell you that the checkout page is slow. Logs tell you what each service said. Neither tells you where the time went when one click fans out to six services. Distributed tracing answers that: it records the path of a single request through every service it touches, with the time spent in each step.

A service mesh sits close to the same problem from the network side. It places a proxy next to every workload to handle encryption, retries and telemetry without changing application code. Tracing and meshes are often discussed together because the mesh's proxies can emit trace data, but they are separate tools and you can use either alone.

Commands and manifests here are read-only: this site has no cluster. The Python examples are runnable stand-ins that implement the core ideas, a span tree and identity-based authorization.

Traces, spans and context

A trace is the story of one request. It is made of spans; each span is one unit of work, such as "handle HTTP request in the cart service" or "SQL query". A span has a name, a start time, a duration and a parent. The first span has no parent and is called the root.

Anatomy of a trace

Trace One request, end to end. Has one trace ID.
Span One step with a start and a duration.
Parent link Each span points to the span that caused it.

Together they form a tree you can read like a flame graph.

1 / 2

How do separate services agree that their spans belong together? Through context propagation: the caller passes a small header to the callee. The W3C standard header is traceparent:

TEXT
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             โ”‚  โ””โ”€โ”€ trace ID (32 hex) โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”” parent span ID โ”˜ โ”” flags
             โ”” version

The callee reads it, creates a new span whose parent is the received span ID, and passes an updated header to the next hop. If one service forgets to forward the header, the trace breaks into two pieces. That is the most common tracing problem, and it is why libraries do this automatically for HTTP clients.

OpenTelemetry

OpenTelemetry (OTel) is the vendor-neutral standard for traces, metrics and logs. It defines the data model, the client SDKs for each language and a wire protocol called OTLP. You instrument your code once and choose a back end later: Jaeger, Grafana Tempo, Zipkin or a commercial service.

In Kubernetes the usual shape is:

  • Your app uses the OTel SDK (or auto-instrumentation) and sends OTLP to a nearby endpoint.
  • An OpenTelemetry Collector runs as a DaemonSet or Deployment (DaemonSets). It receives, batches, filters and forwards telemetry, so apps do not need back-end credentials.
  • A trace store keeps the data, and a UI such as Jaeger or Grafana lets you search it.

A typical pod only needs an environment variable or two:

YAML
env:
  - name: OTEL_SERVICE_NAME
    value: cart
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://otel-collector.observability:4318
  - name: OTEL_TRACES_SAMPLER
    value: parentbased_traceidratio
  - name: OTEL_TRACES_SAMPLER_ARG
    value: "0.1"

The last two lines use sampling: keep 10% of traces, because storing every trace of a busy system is expensive. "Parent-based" means that if the first service decided to keep a trace, everyone downstream keeps it too, so you never get half a trace.

Building a span tree

Trace back ends receive spans in any order, often out of order, and assemble the tree from the parent links. The stand-in below does exactly that and prints a waterfall-style view with each span's own time (its duration minus its children's).

Python Assemble a trace from spans

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading the tree: the gateway span takes 120 ms but only 15 ms of that is its own work, because the rest is spent waiting for cart and payments. The biggest self time is the call to the card network, which is where an engineer would look first. Real trace UIs show the same data as a timeline and handle missing spans by marking a "gap" when a parent was never received.

What a service mesh adds

A service mesh is infrastructure for service-to-service traffic. A data plane of proxies handles the traffic, and a control plane configures them. The two best-known open-source meshes are Istio (feature-rich, based on the Envoy proxy) and Linkerd (smaller and simpler, with its own Rust proxy).

What a mesh gives you

mTLS Encrypted, authenticated pod-to-pod traffic.
Resilience Retries, timeouts and circuit breaking.
Traffic control Canary splits and fault injection.
Telemetry Uniform metrics and traces.
1 / 2

mTLS and identity

With plain TLS only the client checks the server. With mutual TLS (mTLS) both sides present certificates, so each knows who the other is. A mesh automates this: the control plane acts as a certificate authority, gives each workload a short-lived certificate tied to its service account (service accounts) and rotates it automatically. Identities usually look like a SPIFFE ID, for example spiffe://cluster.local/ns/shop/sa/cart.

Because identities are verified, you can write rules by who is calling rather than by IP address. In Istio:

YAML
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-allow-cart
  namespace: shop
spec:
  selector:
    matchLabels:
      app: payments
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/shop/sa/cart"]
      to:
        - operation:
            methods: ["POST"]
            paths: ["/charge"]

The stand-in below checks a caller identity against such a policy. In a real mesh, the proxy does this after verifying the certificate in the TLS handshake. Here we just compare strings, so the cryptography is not simulated.

Python Identity-based authorization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only two of the four calls pass. Note that this protects the application layer: network policies work at the IP and port level and can be combined with it.

Sidecars, ambient and when not to use a mesh

The classic design injects a sidecar proxy into every pod. It is powerful but has costs: extra memory and CPU for every pod, added latency on each hop, a restart needed to upgrade proxies, and more things to debug. See init containers and sidecars for how the pattern works.

Sidecarless designs move the work out of the pod. In Istio's ambient mode, a node-level agent provides mTLS and basic identity for every pod, and an optional shared waypoint proxy adds the heavy layer-7 features only for the namespaces that need them. Linkerd and Cilium also offer lighter-weight approaches.

A mesh is not free, and many teams do not need one. Consider whether you really need it:

  • You may not need it if you have a handful of services, a single team, and TLS to your database is enough. A library-level retry and OpenTelemetry already cover a lot.
  • It fits when you must have mTLS everywhere for compliance, run dozens of services in several languages, or want uniform canary and retry policies without touching code.
  • Be careful with the operational load: a mesh is one more system that can break, and its control plane becomes critical.

Do not adopt a mesh only because it is popular. Add it when you can name the specific problem, such as "we must encrypt all in-cluster traffic and prove service identity", and start with one namespace.

What is real and what is a stand-in

The traceparent format, OTel environment variables and the Istio policy are real syntax. The span-tree builder reproduces how back ends reconstruct a trace but ignores clock skew, sampling and missing parents. The authorization check compares identities that a real mesh would first prove with certificates.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What links the spans of one request across different services into a single trace?

    Each service reads the incoming context, creates a child span and forwards the updated context. The shared trace ID and parent IDs let a back end rebuild the tree.
  2. 2. Why is sampling used in distributed tracing?

    Busy systems create huge numbers of spans. Sampling keeps a percentage, and parent-based sampling keeps traces whole.
  3. 3. What does mutual TLS add compared with ordinary TLS?

    In mTLS both sides present certificates. A mesh uses this to give every workload a verified identity that policies can refer to.
  4. 4. Which is a good reason to skip a service mesh?

    A mesh adds resources, latency and a complex system to operate. With few services the simpler tools are often enough.

Summary

  • A trace is a tree of spans linked by parent IDs; context propagation (traceparent) connects services.
  • OpenTelemetry is the vendor-neutral standard: instrument once, send OTLP to a Collector, store in Jaeger or Tempo, and sample to control cost.
  • A service mesh moves mTLS, retries, traffic splitting and telemetry into proxies configured by a control plane.
  • mTLS gives each workload a verified identity, so policies can say who may call what.
  • Sidecarless meshes cut per-pod overhead; but many small systems are better off without a mesh.

Next: Troubleshooting Playbook, a step-by-step guide to the most common failures.