Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท Resilience and Policies

Istio Circuit Breaking and Outlier Detection Explained

Protect services from overload with Istio circuit breaking: DestinationRule connectionPool limits, outlierDetection ejection, how to read UO and UH errors, and tuning guidelines.

When one service in a chain becomes slow or unhealthy, callers pile up waiting for it, run out of threads and fail too. This is how a small problem becomes a cascading failure. A circuit breaker stops the spread: it limits how many requests may be waiting on a struggling service and temporarily stops sending traffic to instances that keep failing.

In Istio, circuit breaking has two parts, both configured in a DestinationRule: connection pool limits (a cap on concurrency) and outlier detection (ejecting bad endpoints). The product cannot run on this site, so the YAML is read-only and the Python stand-ins simulate both mechanisms.

Connection pool limits

The connectionPool settings cap how much load one client proxy may put on a destination:

YAML
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews-breaker
spec:
  host: reviews
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 20
        connectTimeout: 2s
      http:
        http1MaxPendingRequests: 10
        http2MaxRequests: 100
        maxRequestsPerConnection: 50
        maxRetries: 3
  • tcp.maxConnections: simultaneous TCP connections from this client proxy to the destination (all endpoints together).
  • http.http1MaxPendingRequests: requests that may queue while waiting for a free connection (HTTP/1.1).
  • http.http2MaxRequests: concurrent requests to the destination across HTTP/2 connections.
  • http.maxRequestsPerConnection: after this many requests a connection is recycled; 1 disables keep-alive.
  • http.maxRetries: concurrent retries allowed at one time.

When a limit is exceeded, the client's proxy answers immediately with 503 and the response flag UO (upstream overflow). That is the breaker "opening": fast failure instead of a long wait. Callers should handle it with a fallback or a clear error.

Limits apply per client proxy. If you have 10 client pods and each may open 20 connections, the destination can see up to 200.

Outlier detection

Connection limits protect against volume. Outlier detection protects against bad instances. The proxy watches the responses from each endpoint and removes an endpoint from the load-balancing pool when it fails too often:

YAML
    outlierDetection:
      consecutive5xxErrors: 5
      consecutiveGatewayErrors: 3
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
      minHealthPercent: 30
  • consecutive5xxErrors: how many 5xx responses in a row eject an endpoint.
  • consecutiveGatewayErrors: same, counting only 502, 503, 504 and connection failures.
  • interval: how often the analysis runs.
  • baseEjectionTime: the minimum ejection time; each repeated ejection multiplies it (ejected for baseEjectionTime x number of ejections).
  • maxEjectionPercent: the largest share of endpoints that may be ejected, so you never eject everything.
  • minHealthPercent: below this share of healthy endpoints, outlier detection is paused.

After the ejection time, the endpoint returns to the pool and is watched again. If every endpoint is ejected, or none is ready, clients see 503 with UH (no healthy upstream).

stateDiagram-v2
  [*] --> InPool
  InPool --> Ejected: 5 consecutive 5xx
  Ejected --> InPool: after baseEjectionTime x ejections
  Ejected --> Ejected: ejected again, longer next time

An ejected endpoint is not deleted; it is on the bench for a growing penalty.

Tuning guidelines

  1. Start from measured behavior. A limit below normal peak concurrency causes outages by itself.
  2. Set maxEjectionPercent below 100 so some capacity always remains (the default is 10).
  3. Use outlier detection with retries: a retried request can reach another endpoint while the bad one is being ejected.
  4. Alert on UO and UH rates; a rising count means the protection is working and something upstream needs attention.
  5. Test with a load generator (fortio is commonly used with Istio) and watch the UO flags appear as you raise concurrency.

Try it: a bounded pending queue

Python Connection pool overflow simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Try it: ejection with growing penalty

Python Outlier detection timeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which response flag appears when a connection pool limit is exceeded?

    UO means upstream overflow, the circuit breaker is open for that request.
  2. 2. What does maxEjectionPercent protect against?

    It caps the share of endpoints that outlier detection may remove at once.
  3. 3. Where are circuit breaking settings configured?

    connectionPool and outlierDetection are part of a DestinationRule's trafficPolicy.
  4. 4. Limits such as maxConnections apply...

    Each client sidecar enforces its own limit, so the total load depends on the number of clients.

Summary

  • Circuit breaking in Istio = connectionPool limits (fail fast with UO) + outlierDetection (eject bad endpoints, UH if none left).
  • Both live in a DestinationRule.
  • Limits are per client proxy; tune from measured load.
  • Combine with retries and keep maxEjectionPercent below 100.

Next: Timeouts, retries and retry budgets