Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 36 ยท Resilience and Policies

Istio Timeouts and Retries: Avoiding Retry Storms

Configure timeouts and retries in Istio VirtualServices, understand perTryTimeout and retryOn, calculate retry amplification across layers and design safe deadlines for service chains.

Networks fail in small ways all the time: a pod restarts, a connection resets, a request takes too long. Two simple tools handle most of them: a timeout (stop waiting) and a retry (try again). Istio lets you apply both outside the application code. They are also easy to misuse. Retries at several layers multiply, and a timeout that is too long hides a failure while one that is too short creates one. This lesson shows how to set them correctly.

The product cannot run on this site: configuration is read-only and the Python stand-ins do the arithmetic.

Timeouts in a VirtualService

YAML
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts: [reviews]
  http:
    - route:
        - destination: { host: reviews }
      timeout: 3s

timeout is the total time the proxy waits for a response, including all retries. When it expires, the client receives a 504 with response flag UT. By default Istio sets no route timeout for most traffic (the effective value is effectively unlimited), so always set one for requests initiated by a user.

Retries

YAML
      retries:
        attempts: 2
        perTryTimeout: 1s
        retryOn: connect-failure,refused-stream,unavailable,gateway-error,reset
        retryRemoteLocalities: true
  • attempts: retries in addition to the first try.
  • perTryTimeout: how long each attempt may take. It should be smaller than timeout, otherwise the overall timeout ends the first attempt and retries never happen.
  • retryOn: which failures to retry, using Envoy's conditions: 5xx, gateway-error (502, 503, 504), connect-failure, reset, refused-stream, retriable-4xx, retriable-status-codes and more.
  • retryRemoteLocalities: allow retrying in a different zone.

Even without a retries block, Istio retries twice by default on a small set of connection-level failures; set attempts: 0 to turn retries off on a route.

Only retry operations that are idempotent: reading data, or writes protected by an idempotency key. Retrying a non-idempotent POST (charge a card) can duplicate the effect. Be especially careful with retryOn: 5xx, since a 500 may be returned after the work was done.

Envoy adds exponential backoff with jitter between attempts (base interval 25 ms by default) so retries do not all arrive at the same instant.

The retry storm problem

Suppose a request passes through three services in a chain, A calls B calls C, and each layer is configured with 3 attempts. If C is down, C receives 3 x 3 = 9 requests for every one request arriving at A. With ten times more layers the multiplier explodes. The extra load lands on a service that is already struggling, which makes recovery harder. This is a retry storm.

flowchart LR
  U["1 user request"] --> A["A: 3 attempts"]
  A --> B["B: 3 attempts each"]
  B --> C["C (failing): receives 9"]

Retries multiply along the call chain.

Mitigations:

  1. Retry in one layer only, usually the one closest to the user, or at the edge.
  2. Keep attempts small (1 or 2).
  3. Use retry budgets: Envoy supports limiting concurrent retries as a fraction of active requests; in Istio the maxRetries field of the connection pool caps simultaneous retries to a destination (see the circuit-breaking lesson, Circuit breaking).
  4. Pair retries with outlier detection so retries land on other endpoints.

Deadlines across a chain

Each hop's timeout should be smaller than its caller's, leaving room for retries. If A has a 3 second timeout, and B (called by A) takes up to 2 seconds with two attempts, A will time out before B's retry can matter. A simple rule: the caller's timeout is at least attempts x perTryTimeout of its callee plus processing time.

Try it: retry amplification

Python How many requests does the failing service see?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try it: timeout and retry planner

Python Does the overall timeout fit the retries?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fault injection for testing

To prove your settings work, inject a failure rather than waiting for one. A VirtualService fault block can add a fixed delay or return an abort for a percentage of requests, then you watch the timeouts, retries and breakers react:

YAML
      fault:
        delay:
          percentage: { value: 50 }
          fixedDelay: 5s
        abort:
          percentage: { value: 10 }
          httpStatus: 503

Try it in a test namespace first, never silently in production.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Does the VirtualService timeout include retries?

    timeout is overall; perTryTimeout limits each individual attempt.
  2. 2. Three layers each use 3 attempts (retries). How many requests reach a failing last service per user request?

    Each layer makes 1 + 3 = 4 tries, so the multiplier is 4 x 4 x 4 = 64.
  3. 3. What is the problem with retrying a non-idempotent POST on 5xx?

    A 5xx can arrive after the work was done; retrying duplicates the effect unless protected by an idempotency key.
  4. 4. What response and flag do callers get when the route timeout expires?

    A timeout produces 504 with the UT flag.

Summary

  • Always set a route timeout; it covers all attempts.
  • perTryTimeout must be smaller than timeout; choose retryOn conditions carefully.
  • Retries multiply across layers: retry in one place and keep attempts small.
  • Retry only idempotent calls; cap concurrent retries and combine with outlier detection.

Next: Rate limiting in Istio