Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 36 ยท Observability

Istio Metrics and Prometheus: The Golden Signals for Free

Learn the standard Istio metrics (istio_requests_total, duration, bytes), their labels, how Prometheus scrapes proxies, how to customize metrics with the Telemetry API and useful PromQL queries.

Once every request flows through an Envoy proxy, the mesh can count, time and label all of them without a single line of application code. That is the biggest day-one payoff of Istio: uniform request rate, error rate and latency for every service, whether it is written in Go, Java or Python. In this lesson you learn which metrics the proxies emit, what their labels mean, how Prometheus collects them and how to ask useful questions with PromQL.

The Istio product itself (a Kubernetes cluster with Envoy sidecars) cannot run on this site. Commands and YAML are therefore read-only, and the runnable examples are small Python stand-ins that parse and compute the same things you will do with real metrics.

The standard metrics

Every sidecar (and every gateway) exposes the same set of standard metrics. These are the ones you will use daily:

Metric Type Meaning
istio_requests_total counter Number of HTTP/gRPC requests handled
istio_request_duration_milliseconds histogram Request latency as seen by the proxy
istio_request_bytes histogram Size of request bodies
istio_response_bytes histogram Size of response bodies
istio_tcp_connections_opened_total counter TCP connections opened
istio_tcp_sent_bytes_total / istio_tcp_received_bytes_total counter TCP bytes in each direction

A single request produces two data points: one from the client-side proxy (reporter="source") and one from the server-side proxy (reporter="destination"). Comparing them tells you whether a problem lives in the network or in the service.

Labels (dimensions)

Metrics are only useful because of their labels. The most important ones:

  • source_workload, source_workload_namespace, source_app: who called.
  • destination_workload, destination_service, destination_service_name, destination_version: who was called.
  • request_protocol: http, grpc, tcp.
  • response_code: the HTTP status (or 0 when no response arrived).
  • response_flags: Envoy's short reason codes such as UH (no healthy upstream) or - for none. You will decode them in the access-logs lesson, Access logs.
  • connection_security_policy: mutual_tls or none. This makes it easy to prove which traffic is encrypted.
  • reporter: source or destination.

More labels mean more time series. Adding a label with many distinct values (a user id, a full URL path) can overwhelm Prometheus. Keep dimensions low-cardinality.

How Prometheus gets the numbers

Each pod's istio-proxy container merges Envoy statistics, Istio's standard metrics and, optionally, the application's own metrics, and serves them at :15020/stats/prometheus. A Prometheus server scrapes that endpoint. Istio ships a ready-made demo Prometheus in its samples/addons folder (not meant for production), and production clusters usually use their own Prometheus or an operator such as the Prometheus Operator.

Terminal
kubectl apply -f samples/addons/prometheus.yaml
istioctl dashboard prometheus      # port-forwards and opens the UI

Useful PromQL queries

PROMQL
# Requests per second to the reviews service, by response code
sum(rate(istio_requests_total{reporter="destination", destination_service_name="reviews"}[5m])) by (response_code)

# Error ratio (5xx) for the same service
sum(rate(istio_requests_total{reporter="destination", destination_service_name="reviews", response_code=~"5.."}[5m]))
/
sum(rate(istio_requests_total{reporter="destination", destination_service_name="reviews"}[5m]))

# p99 latency in milliseconds
histogram_quantile(0.99,
  sum(rate(istio_request_duration_milliseconds_bucket{reporter="destination", destination_service_name="reviews"}[5m])) by (le))

Always pick one reporter. If you sum both, every request is counted twice.

Try it: parse and summarize metrics

Prometheus exposes metrics as plain text. This stand-in parses a small sample of that format and computes a success rate per destination, which is what the second query above does.

Python Parse Prometheus text and compute error ratio

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Customizing metrics with the Telemetry API

The Telemetry resource lets you remove, add or change dimensions without touching the proxy configuration by hand. This example removes a label mesh-wide for one metric to reduce cardinality, and adds a custom dimension taken from a request header.

YAML
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: trim-request-metrics
  namespace: istio-system        # root namespace = mesh-wide
spec:
  metrics:
    - providers:
        - name: prometheus
      overrides:
        - match:
            metric: REQUEST_COUNT
            mode: CLIENT_AND_SERVER
          tagOverrides:
            request_protocol:
              operation: REMOVE
            tenant:
              operation: UPSERT
              value: "request.headers['x-tenant'] | 'unknown'"

Settings in the root namespace apply to the whole mesh, a resource in an application namespace overrides them for that namespace, and a selector narrows it to specific workloads. Note that custom dimensions must also be registered with the proxy (through sidecar.istio.io/extraStatTags) before Prometheus can see them.

Try it: estimate a latency percentile from histogram buckets

Prometheus histograms store cumulative bucket counts. histogram_quantile interpolates inside a bucket. This stand-in reproduces that arithmetic.

Python histogram_quantile by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimate is only as precise as the buckets. If your SLO is 200 ms, make sure there is a bucket boundary near 200.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which label tells you whether a data point was measured by the caller's proxy or the callee's proxy?

    The reporter label is source or destination. Using both at once double counts requests.
  2. 2. Where does Prometheus scrape merged Istio and Envoy metrics from a pod?

    The sidecar agent merges metrics and serves them on 15020 so Prometheus needs only one endpoint per pod.
  3. 3. Which metric should you use for error ratio queries?

    istio_requests_total is a counter with a response_code label, so a ratio of 5xx to all requests gives the error rate.
  4. 4. Why should you avoid putting a user id into a metric label?

    High-cardinality labels multiply the number of time series and memory use.

Summary

  • Sidecars emit standard metrics for HTTP, gRPC and TCP with no code changes.
  • istio_requests_total plus istio_request_duration_milliseconds give you rate, errors and latency.
  • Filter on one reporter to avoid double counting.
  • Prometheus scrapes :15020/stats/prometheus on each pod.
  • The Telemetry API adds, removes or overrides dimensions; keep cardinality low.

Next: Distributed tracing in Istio