Skip to content
Tutorials Center
My learning

Topics

Lesson 129 of 147 ยท Advanced Go

Retries, Timeouts and Circuit Breakers

Make Go services survive flaky dependencies using only the standard library - deadlines, retries with exponential backoff and jitter, Retry-After handling and a circuit breaker state machine.

Networks drop packets, servers restart, databases hiccup and a partner's API has a bad afternoon. In a system made of many services, something is always failing a little. The difference between a fragile program and a resilient one is what happens next: does one slow dependency freeze everything that calls it, or does your service notice, cope, and recover on its own?

Resilience comes from a few small habits that fit together. Put a timeout on every call so nothing waits forever. Retry failures that are likely to be temporary, but slowly and with randomness, so you do not make an outage worse. Use a circuit breaker to stop calling a dependency that is clearly down, giving it room to recover. In this lesson you will write all three with nothing but the standard library, make them testable with an injected clock, and see how they combine. Two earlier lessons pair well with this one: context for deadlines and rate limiting and backpressure for protecting capacity.

flowchart LR
    A["Your handler"] --> B["Circuit breaker<br/>(fail fast when open)"]
    B --> C["Retry loop<br/>(backoff + jitter)"]
    C --> D["Per-attempt timeout"]
    D --> E["Dependency"]

Resilience layers around one outgoing call. Each layer solves a different problem.

Timeouts come first

A call without a timeout can block forever. In a server, each stuck call holds a goroutine, a connection and memory, until the whole service is stuck too. Always set two kinds of limit:

  • An overall deadline for the work the user is waiting for, passed down as a context.Context.
  • A per-attempt timeout shorter than that, so one hung attempt leaves time for a retry.

The default http.Client has no timeout, which surprises many newcomers. Set Client.Timeout for the whole exchange (connect, redirects and reading the body), or better, build each request with http.NewRequestWithContext and a context with a deadline. Servers need timeouts too; see building an HTTP server and the HTTP client.

Go
client := &http.Client{Timeout: 5 * time.Second} // simple, applies to every request

// or per request:
ctx, cancel := context.WithTimeout(parent, 800*time.Millisecond)
defer cancel()
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
resp, err := client.Do(req)

When the deadline passes, the call returns an error that satisfies errors.Is(err, context.DeadlineExceeded), and the request is cancelled on the wire.

Retries with exponential backoff and jitter

Some failures are temporary: a connection reset, a 503, a lock timeout. Trying again often works. But retrying badly is dangerous. If a database is struggling and a thousand clients retry instantly, three times each, you have tripled the load on the thing that is already failing. Good retries follow rules:

  1. Retry only what is retryable. Network errors, timeouts, 429 and 5xx are candidates. A 400 or 404 will fail the same way every time, so give up at once.
  2. Limit the attempts. Three to five is typical. Never retry forever.
  3. Back off exponentially. Wait 100 ms, then 200 ms, then 400 ms, up to a maximum.
  4. Add jitter. If all clients wait exactly the same time, they retry together in waves. Randomizing the wait ("full jitter" picks a random value between 0 and the delay) spreads them out.
  5. Honor Retry-After. If the server says when to come back, listen.
  6. Respect the context. Stop retrying when the caller's deadline passes.
  7. Only retry operations that are safe to repeat (idempotent). Repeating GET is fine. Repeating a payment POST could charge twice unless the API supports an idempotency key.

Generics make a reusable helper easy. The function below takes the work as a callback, wraps errors that must not be retried in Permanent, and lets an error carry its own RetryAfter. The sleep function is injectable, so the first example can run instantly and print the schedule it would follow.

Go A generic Retry helper with backoff and jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The first run shows the waits [100ms 200ms 400ms] before success on attempt 4. The second shows the doubling stopping at the Max of one second. With a permanent error the function returns after one call and sleeps zero times. Note %w in the final error: callers can still use errors.Is or errors.As on the last real failure (see wrapping errors).

Check how the generic helper is used: the type parameter T is inferred from the callback's return type, so one function serves HTTP calls, database queries and anything else. Retrying a function that has side effects needs care in the callback, not in the helper.

Retrying real HTTP calls

Now connect the helper to a real round trip. The test server below misbehaves on purpose: its first reply hangs, the second says 429 with Retry-After: 1, and the third works. The client uses a short per-attempt timeout, treats network errors, 429 and 5xx as retryable, treats other 4xx as permanent, and rebuilds the request for every attempt, because a request body can be read only once.

Go Retrying a flaky HTTP endpoint

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Read the log lines: attempt 1 times out after 100 ms (the server hung), attempt 2 gets 429 and the client waits the full second the server asked for, and attempt 3 succeeds. The unknown path produces a 404 and stops at once. In the last case the caller's 30 ms deadline fires while attempt 1 is in flight, and the whole fetch ends with a deadline error instead of burning through retries nobody is waiting for.

Backoff protects the dependency from you. Add a retry budget at scale: for example, allow retries to be at most 10 percent of normal traffic, so a widespread failure cannot multiply load. And if a dependency is truly down, retries only add delay for your users. That is where the circuit breaker helps.

The circuit breaker

An electrical breaker trips when current is too high, protecting the wiring. A software circuit breaker does the same for calls to a failing service: after enough consecutive failures it "opens", and further calls fail immediately without touching the dependency. After a cooling-off period it lets a single trial call through. If that works, the breaker closes and traffic resumes; if not, it stays open for another round.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: failures reach the threshold
    Open --> HalfOpen: cooldown has passed
    HalfOpen --> Closed: trial call succeeds
    HalfOpen --> Open: trial call fails

The three states. Closed is normal; Open fails fast; Half-open allows one probe.

This does two things at once. Your callers get a fast, predictable error instead of waiting for a timeout, and the struggling service gets quiet time to recover. The implementation is a small state machine protected by a mutex. It uses an injected clock, so you can test the cooldown without waiting.

Go A circuit breaker state machine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

The trace shows the whole life cycle. Three failures open the breaker. The next two calls fail instantly with ErrOpen and, as the last column proves, never reach the dependency. After the cooldown one trial goes through; it fails, so the breaker opens again. After the next cooldown the dependency is back, the trial succeeds and the breaker closes.

Design choices to think about when you write a breaker for real: count failures in a time window or as a percentage rather than strictly consecutive; decide which errors count (a 404 is not a sign of an outage, a timeout is); allow a few trial calls in half-open state; and expose the state as a metric so you can see it on a dashboard. Libraries such as sony/gobreaker implement these options, but the core is the small state machine above.

Putting the layers together

The order of the layers matters. The usual arrangement puts the breaker outside the retry loop, so a persistent failure trips it once rather than after every retry, and treats ErrOpen as permanent so you do not retry a call the breaker just refused:

Go
func (c *Client) Get(ctx context.Context, url string) (string, error) {
    return c.breaker.Do(func() error { // fail fast when the dependency is known to be down
        var err error
        body, err = Retry(ctx, c.retryCfg, func(ctx context.Context, n int) (string, error) {
            return c.once(ctx, url, n) // per-attempt timeout lives inside
        })
        return err
    })
}

Other tools round out the toolbox. A fallback returns a cached or default value when the real call fails (show yesterday's recommendations instead of an error page). A bulkhead gives each dependency its own concurrency limit, as in the semaphore example, so one slow service cannot use up all your goroutines. Hedged requests send a second copy of a slow read after a short delay and take whichever answers first, trading extra load for lower latency. All of them share a rule: decide in advance what failure should look like to the user, and test it.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why add random jitter to exponential backoff?

    Without jitter, every client that failed at the same moment retries at the same moments, producing repeated spikes on a recovering service. Randomizing spreads the retries out.
  2. 2. Which response should a client normally NOT retry?

    A 400 means the request itself is wrong, so repeating it gives the same answer. 503, 429 and connection errors are often temporary and worth another try.
  3. 3. In the retry helper, why is the HTTP request built inside the attempt instead of once before the loop?

    A consumed body cannot be re-sent, and the per-attempt context carries its own deadline. Rebuilding the request for each attempt avoids both problems.
  4. 4. A circuit breaker is open. What happens when a new call arrives before the cooldown ends?

    Failing fast is the whole point. The caller gets an instant error and the failing service is left alone to recover.
  5. 5. Why should a retry loop watch the caller's context?

    When the caller's deadline has passed or the request was cancelled, more retries serve no one. Sleeping with a select on ctx.Done() lets the loop exit right away.

Summary

  • Put a timeout on every call: an overall deadline from the caller's context and a shorter per-attempt timeout. http.Client has no timeout by default.
  • Retry only temporary, safe-to-repeat failures. Limit attempts, back off exponentially, add jitter, honor Retry-After, and stop when the context ends.
  • Mark non-retryable errors as permanent, rebuild requests for each attempt, and wrap the last error with %w.
  • A circuit breaker is a closed / open / half-open state machine that fails fast against a failing dependency and probes it after a cooldown. Inject the clock to test it.
  • Layer them as breaker, then retry, then per-attempt timeout, and add fallbacks and bulkheads where the user experience calls for them.

Next: Project: Command-Line To-Do Manager