Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 40 ยท Timeouts and Cancellation

Retries, Backoff and Circuit Breakers in asyncio

Write reliable async retry logic: exponential backoff with jitter, per-attempt and total timeouts, retrying only safe errors, hedged requests and a simple circuit breaker.

Networks fail in small, temporary ways: a dropped packet, a server restarting, a rate limit that clears in a second. A request that failed once often succeeds a moment later, so wrapping calls in a retry loop is one of the cheapest reliability wins you can get. But careless retries also turn a small outage into a big one, because thousands of clients hammering a struggling server make it worse.

In this lesson you will build a retry helper step by step, add exponential backoff and random jitter, combine it with timeouts, and finish with two related patterns: hedged requests and a small circuit breaker. All the examples use a fake flaky service so the output is repeatable.

The simplest retry

Python
for attempt in range(1, 4):
    try:
        return await call()
    except ConnectionError:
        if attempt == 3:
            raise

This has two problems: it retries instantly (a server that just failed will probably fail again right away), and it retries every time in lock-step with every other client. Both are fixed by adding a delay that grows.

Exponential backoff with jitter

Exponential backoff waits longer after each failure: 0.1 s, 0.2 s, 0.4 s, 0.8 s and so on, up to a cap. Jitter adds randomness so a crowd of clients that failed together do not all retry at the same instant (the "thundering herd"). A popular recipe is full jitter: sleep a random time between zero and the backoff ceiling.

flowchart TD
    A["Call"] --> B{"Success?"}
    B -->|"yes"| Z["Return result"]
    B -->|"no, retryable"| C{"Attempts left?"}
    C -->|"no"| Y["Raise last error"]
    C -->|"yes"| D["Sleep random 0..min(cap, base * 2^n)"]
    D --> A
    B -->|"no, permanent error"| Y

A retry loop retries only errors that might go away, and waits a growing, randomised time between attempts.

Here is a reusable helper. It takes a factory (a function returning a fresh coroutine each time), because a coroutine object can only be awaited once.

Python A retry helper with backoff and jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The helper only retries the exception types you list in retry_on. A ValueError (a bug or bad request) is raised at once, because retrying cannot fix it. This distinction between transient and permanent errors is the most important design decision in any retry logic. Typical transient ones: connection resets, timeouts, HTTP 429/502/503/504. Typical permanent ones: HTTP 400/401/403/404, validation errors.

Per-attempt timeouts and a total budget

Without a timeout, one stuck attempt blocks the loop for as long as the server likes. Give each attempt its own deadline, and give the whole operation an overall budget, so the caller is never kept waiting beyond a known limit:

Python Per-attempt timeout inside a total deadline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The inner timeout cuts off each hung attempt; the outer one guarantees the caller waits at most one second no matter how many attempts remain. Choose the budget first, then divide it among the attempts.

Rules for safe retries

  • Retry only idempotent operations, or operations you made idempotent with a request ID. Retrying "charge the card" after a timeout may charge twice, because the first attempt might have succeeded.
  • Cap everything: attempts, delay and total time. Unbounded retry loops hide outages.
  • Respect Retry-After headers and rate-limit responses; sleep at least that long.
  • Log each retry with the reason, so a flapping dependency is visible.
  • Don't retry in layers. If three layers each retry three times, one failure becomes 27 calls. Retry at one level only.
  • Let asyncio.CancelledError pass through. Because it is not an Exception, the except retry_on clauses above never catch it, which is what you want.

Hedged requests

A hedged request attacks slow responses rather than failures: send the request, and if there is no answer after a short delay, send a duplicate to another server and take whichever answers first. It trims the long tail of latency at the price of some extra load, and it is safe only for idempotent reads.

Python Hedging with FIRST_COMPLETED

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A tiny circuit breaker

If a dependency is completely down, retrying just burns time. A circuit breaker notices repeated failures, "opens" (fails instantly for a cool-down period), then lets a trial request through to see whether the service has recovered.

Python Circuit breaker with a cool-down

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Requests 3 and 4 fail instantly without touching the service, which protects both the caller's latency and the struggling server. Production libraries add a proper half-open state, metrics and per-endpoint breakers, but the idea is exactly this.

For limiting how many calls are in flight at once, see rate limiting and concurrency limits.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why add random jitter to a backoff delay?

    Clients that failed together would otherwise retry together, hitting the server in synchronised waves. Randomness spreads them out.
  2. 2. Why does a retry helper take a factory function instead of a coroutine object?

    Awaiting an already-used coroutine raises RuntimeError. The factory creates a new coroutine for every attempt.
  3. 3. Which operation is the most dangerous to retry blindly after a timeout?

    The first attempt may have succeeded before the timeout, so a retry could charge twice. Use idempotency keys for such operations.
  4. 4. What is the purpose of a circuit breaker?

    After repeated failures the breaker opens and rejects calls immediately for a cool-down period, protecting latency and the failing service.

Summary

  • Retry only transient, idempotent failures; raise permanent errors immediately.
  • Back off exponentially with a cap, add jitter, and bound attempts and total time.
  • Combine a per-attempt asyncio.timeout() with an overall budget.
  • Hedged requests cut tail latency; circuit breakers stop you hammering a dead dependency.

Next: Async iteration and generator patterns