Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 39 ยท Error Handling and Metadata

gRPC Retries, Backoff and Hedging

Configure gRPC retry policies with service config, understand exponential backoff, retry throttling, idempotency and hedging, with runnable Go simulations.

Networks fail in small, temporary ways: a pod restarts, a connection resets, a load balancer drains. A client that retries sensibly turns most of these blips into invisible hiccups. A client that retries carelessly turns a small outage into a big one, because every caller piles extra requests onto a service that is already struggling.

This lesson covers what gRPC offers: transparent retries done by the library, configured retries from a service config, hedging, and the rules that keep them safe.

The real grpc-go library cannot run on this site. JSON service config and Go options are shown read-only. The runnable blocks simulate the algorithms (backoff, throttling, hedging) with plain Go so you can see their behaviour.

Three kinds of retry

  1. Transparent retry. If a call failed before any byte reached the server (for example the connection was refused), gRPC silently retries it, even without a policy. You cannot observe it.
  2. Configured retry. If the call reached the server and failed with a status you listed (usually UNAVAILABLE), the client retries according to a policy.
  3. Hedging. The client sends the same request to several backends, with a small delay between copies, and uses the first good answer. It trades extra load for lower tail latency.

A call that already received response headers is committed and is never retried automatically, because part of the answer may have been consumed.

A retry policy in service config

Retries are configured per method in the service config, a JSON document that comes from the client (via a dial option) or from the name resolver (for example DNS TXT or xDS):

JSON
{
  "methodConfig": [{
    "name": [{ "service": "users.v1.UserService", "method": "GetUser" }],
    "timeout": "2s",
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "1s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}

In Go you pass it when dialing:

Go
conn, err := grpc.NewClient(target,
    grpc.WithTransportCredentials(creds),
    grpc.WithDefaultServiceConfig(serviceConfigJSON),
)

maxAttempts includes the first try, so 4 means one try and up to three retries (the library caps it at 5). The timeout here is the deadline for the whole call including retries; see deadlines.

Exponential backoff with jitter

Waiting before each retry gives the server room to recover. The delay for retry n is random between 0 and min(maxBackoff, initialBackoff * multiplier^(n-1)). The randomness, called jitter, stops thousands of clients from retrying in lockstep.

Go Backoff schedule with jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The ceilings are 100 ms, 200 ms, 400 ms, 800 ms, then capped at 1 s. The actual waits differ from run to run unless the generator is seeded, as it is here.

Retry the right things

Only retry when the same request has a fair chance of succeeding next time:

  • UNAVAILABLE: yes.
  • DEADLINE_EXCEEDED: usually not inside a retry policy, because the time budget is gone.
  • INVALID_ARGUMENT, NOT_FOUND, PERMISSION_DENIED: never; the answer will not change.

And only retry idempotent operations: ones where doing the work twice gives the same result as doing it once. GetUser is naturally idempotent. CreateOrder is not: a retry after a lost response may create two orders. Make such calls safe by adding a client-generated idempotency key to the request and having the server remember results by key.

Go Idempotency key makes a retry safe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Without the key, the lost reply would create a second order. With it, the retry finds the first result.

Retry throttling

If a backend is down, every client retrying three times triples the load at the worst moment. Retry throttling adds a token bucket per client: each failure removes a token, each success adds a fraction of one, and retries are suppressed when the bucket is below half. It lives in service config:

JSON
"retryThrottling": { "maxTokens": 10, "tokenRatio": 0.1 }
Go Token-bucket retry throttling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

After five failures the bucket drops to 5, retries stop, and only a run of successes re-enables them. Servers can also push back: a response trailer grpc-retry-pushback-ms tells the client how long to wait, or -1 to never retry.

Hedging

A hedging policy sends the first request immediately, and further copies after hedgingDelay if no response arrived yet:

JSON
"hedgingPolicy": {
  "maxAttempts": 3,
  "hedgingDelay": "0.05s",
  "nonFatalStatusCodes": ["UNAVAILABLE", "INTERNAL"]
}

The first successful response wins and the others are cancelled. Hedging cuts the slow tail (the p99) but increases load, so use it for idempotent, cheap reads. A method has either a retryPolicy or a hedgingPolicy, never both.

Go Hedging picks the fastest replica

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does maxAttempts: 4 mean in a gRPC retry policy?

    maxAttempts counts the original call too, so 4 means the first attempt and at most three retries.
  2. 2. Why is jitter added to backoff delays?

    Random delays spread retries over time so recovering servers are not hit by synchronized waves.
  3. 3. Which operation needs an idempotency key before it is safe to retry?

    Creating something twice has a different result than once. A key lets the server detect the duplicate.
  4. 4. What is hedging?

    Hedging reduces tail latency by racing staggered copies of the same request, at the cost of extra load.
  5. 5. A call has already received response headers. What happens on failure?

    After headers arrive the call is committed, because the client may already have consumed part of the answer.

Summary

  • gRPC does transparent retries for calls that never left the client, configured retries from service config, and optional hedging.
  • A policy lists maxAttempts, backoff settings and the status codes to retry, mostly UNAVAILABLE.
  • Backoff is exponential with jitter; throttling stops retry storms; servers can push back.
  • Retry only idempotent calls, and use idempotency keys for creates.
  • Hedging lowers tail latency for cheap idempotent reads at the price of extra load.

Next: Interceptors Explained