Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 36 ยท Operations and Upgrades

Troubleshooting Istio: istioctl analyze, proxy-status, proxy-config

A practical debugging workflow for Istio: istioctl analyze, proxy-status, proxy-config, describe and Envoy logs, with the most common causes of 503 errors, 404s, mTLS failures and missing sidecars.

When something breaks in a service mesh, the question is always the same: is it the application, the network, or the mesh configuration? Istio gives you a layered set of tools to answer it. This lesson teaches the workflow, in the order you should use the tools, and ends with a catalog of the most frequent problems and their symptoms.

The product cannot run on this site: commands are read-only, and Python stand-ins implement a small configuration linter and a symptom-to-cause lookup.

Step 1: Is the configuration valid?

Terminal
istioctl analyze -A                 # whole cluster
istioctl analyze -n shop            # one namespace
istioctl analyze my-virtualservice.yaml    # a file, before applying

analyze reads live (or local) Istio configuration and reports problems with codes such as IST0101 (a referenced host or gateway does not exist), IST0106 (schema validation), IST0108 (unknown annotation), IST0118 (service port name does not follow the naming convention), IST0145 (conflicting gateway hosts) and the warning that a namespace is not labeled for injection. Fix what it reports first; it catches an astonishing share of problems in seconds. Run it in CI against your manifests.

Step 2: Are the proxies in sync with the control plane?

Terminal
istioctl proxy-status

Each proxy is listed with its sync state for the discovery services: CDS (clusters), LDS (listeners), EDS (endpoints) and RDS (routes). SYNCED is good; STALE means istiod pushed configuration but the proxy has not acknowledged it (network problem or an overloaded proxy); NOT SENT simply means there was nothing to send. A pod missing from the list is not connected at all, usually because it has no sidecar.

Step 3: What does the proxy actually have?

istioctl proxy-config (alias pc) shows the Envoy configuration the proxy holds, which is the truth about what will happen to a request:

Terminal
istioctl pc listeners deploy/productpage -n shop
istioctl pc routes    deploy/productpage -n shop --name 9080
istioctl pc clusters  deploy/productpage -n shop --fqdn reviews.shop.svc.cluster.local
istioctl pc endpoints deploy/productpage -n shop --cluster "outbound|9080||reviews.shop.svc.cluster.local"
istioctl pc secret    deploy/productpage -n shop         # certificates and their validity

Think of the order as a request's life: listener (port), route (host and path), cluster (service and subset), endpoint (pod IP and health). A missing route means a missing or mismatched VirtualService (response flag NR); a cluster with no healthy endpoints means UH.

Friendlier summaries:

Terminal
istioctl x describe pod productpage-v1-abc -n shop     # which Services, VirtualServices and policies apply to this pod
istioctl x precheck                                     # before installing or upgrading

Step 4: Logs and the Envoy admin interface

Terminal
kubectl logs deploy/reviews -c istio-proxy -n shop --tail=50      # access logs and proxy errors
kubectl logs -n istio-system deploy/istiod                        # control plane errors
istioctl admin log productpage-v1-abc.shop --level connection:debug   # raise log verbosity temporarily
istioctl dashboard envoy deploy/productpage -n shop               # Envoy admin page (config_dump, stats)

Raise log levels briefly; debug logging is very noisy. For a trace of what Envoy decided, combine the response flag in the access log (see Access logs) with the relevant proxy-config output.

Common problems and their symptoms

Symptom Likely cause First check
No sidecar in pod Namespace not labeled, or pod created before the label kubectl get ns --show-labels, restart pods
404 from the gateway VirtualService not bound to the gateway or hosts mismatch analyze, pc routes on the gateway
503 NR No route for host or port VirtualService host, port name
503 UH No healthy endpoints (ejected or not ready) pc endpoints, readiness probes
503 UF or "connection reset" mTLS mode mismatch, wrong port PeerAuthentication and DestinationRule TLS settings
503 UO Circuit breaker limits DestinationRule connectionPool
HTTP features not working Port treated as TCP Service port name (http-...) or appProtocol
403 RBAC: access denied AuthorizationPolicy denies the request List policies in the namespace, check source principal
Config changes not applied Proxy STALE, or resource in the wrong namespace proxy-status, analyze
Slow convergence, high istiod CPU Huge mesh without Sidecar scoping Sidecar scoping

One classic: after enabling STRICT mTLS, a pod without a sidecar (a job, a legacy workload) cannot talk to meshed services and gets connection resets. Use PERMISSIVE mode during migration and watch which clients still send plaintext.

Try it: a mini configuration linter

This stand-in does a few of the checks that istioctl analyze performs, on plain Python dictionaries.

Python Lint a VirtualService against known services

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try it: symptom to cause

Python Diagnose from the access log flag

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the last case: a plain 500 with no flag came from the application, not from Envoy.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which command checks Istio configuration for mistakes such as unknown hosts or wrong port names?

    analyze validates configuration and reports issues with IST codes.
  2. 2. What does a STALE state in proxy-status mean?

    STALE indicates a sync problem between the control plane and that proxy.
  3. 3. In which order does a request meet the Envoy configuration objects?

    Listener (port), route (host and path), cluster (service) and endpoint (pod).
  4. 4. After enabling STRICT mTLS, a pod without a sidecar gets connection resets. Why?

    STRICT mode rejects plaintext connections; migrate with PERMISSIVE first.

Summary

  • Work top-down: analyze, proxy-status, proxy-config, then logs and the admin interface.
  • The response flag separates mesh decisions from application errors.
  • Common culprits: missing injection, mismatched hosts, port naming, mTLS modes, policies and limits.
  • Run istioctl analyze in CI to catch mistakes before they ship.

Next: Multicluster Istio overview