Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 36 ยท Retries and Reliability

Celery Retries and Exponential Backoff

Make Celery tasks resilient: self.retry, autoretry_for, retry_backoff, jitter, max_retries, custom countdowns, and how to avoid retry storms.

Distributed systems fail in small, temporary ways: a database connection drops, an API returns 503, a DNS lookup times out. Most of these problems disappear if you simply try again a little later. Celery has first-class support for this. A task can ask to be re-queued with a delay, and you can describe a whole retry policy in the task decorator.

Celery and its workers do not run on this site, so Celery code is read-only. The runnable examples are plain-Python stand-ins that compute the same delays and follow the same retry logic, with fake (instant) waiting.

self.retry

Add bind=True so the task receives itself as self, then call self.retry() inside an except:

Python
import requests
from celery import Celery

app = Celery("shop", broker="redis://localhost:6379/0")

@app.task(bind=True, max_retries=5, default_retry_delay=30)
def fetch_rates(self, currency):
    try:
        response = requests.get(f"https://api.example.com/rates/{currency}", timeout=5)
        response.raise_for_status()
        return response.json()
    except requests.RequestException as exc:
        raise self.retry(exc=exc, countdown=30)

What self.retry does: it sends a new message for the same task (same id, same arguments, incremented retry counter) with a countdown, then raises a special Retry exception so that the current run stops. Write raise self.retry(...) rather than a bare call so the code after it never runs.

Important settings:

Option Meaning
max_retries How many retries before giving up (default 3). None retries forever.
default_retry_delay Seconds to wait between retries (default 180)
countdown in retry() Overrides the delay for this retry
exc in retry() The exception to raise if retries are exhausted

When max_retries is exceeded, self.retry(exc=exc) re-raises exc, and the task ends in FAILURE. self.request.retries tells you which attempt you are on.

autoretry_for: the declarative way

Writing try/except in every task is repetitive. Declare which exceptions should trigger a retry:

Python
@app.task(
    autoretry_for=(requests.RequestException,),
    retry_backoff=True,
    retry_backoff_max=600,
    retry_jitter=True,
    max_retries=6,
)
def fetch_rates(currency):
    response = requests.get(f"https://api.example.com/rates/{currency}", timeout=5)
    response.raise_for_status()
    return response.json()

Now any RequestException raised inside the function schedules a retry automatically. Other exceptions (a KeyError from a bug) fail immediately, which is what you want: bugs should not be retried 6 times.

You can also pass retry_kwargs={"max_retries": 5, "countdown": 10}, and dont_autoretry_for to exclude subclasses.

Exponential backoff and jitter

With retry_backoff=True the delay doubles each attempt: 1, 2, 4, 8 seconds, and so on (a number like retry_backoff=3 sets the factor so delays are 3, 6, 12...). retry_backoff_max caps the delay (default 600 s). retry_jitter=True (the default) picks a random delay between 0 and the computed value so a thousand failed tasks do not all retry at the same instant.

Why it matters: if a service is overloaded and every client retries every 30 seconds in lockstep, the retries themselves keep it down. Backoff gives it room to recover and jitter spreads the load.

Python Stand-in: compute the backoff schedule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Simulating the full retry loop

The next stand-in plays a flaky API that fails twice, then succeeds, and a broken one that never recovers. The retry decorator below is not Celery; it reproduces the control flow so you can see attempts, delays and the final failure.

Python Stand-in: retries, success and giving up

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Retry rules of thumb

  1. Retry only transient errors: network timeouts, 5xx, lock contention, deadlocks. Never retry validation errors or bugs.
  2. Always use backoff with jitter for calls to shared services.
  3. Set a finite max_retries and a sensible retry_backoff_max. A task retrying forever hides an outage.
  4. Make the task idempotent, because a retry may repeat work that half-succeeded. See Acks late and idempotency.
  5. Mind the visibility timeout with Redis or SQS brokers: a task with a long countdown or eta held by a worker may be redelivered if the delay exceeds the broker's visibility timeout.
  6. Alert on exhausted retries with an error callback (Failure handling), so someone learns the job is dead.
  7. Prefer retry over time.sleep. Sleeping in a task blocks a worker slot; a retry frees it.

Retrying a different way: self.replace and re-queue

Sometimes a retry should go to another queue (for example a slow "retry" queue so failing jobs do not block fresh ones):

Python
raise self.retry(exc=exc, countdown=60, queue="retry")

Keyword arguments to retry() are the same as apply_async options.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why write raise self.retry(...) instead of just self.retry(...)?

    retry() sends the new message and raises Retry. Writing raise makes sure no code after it runs and is clear to readers and linters.
  2. 2. What does retry_backoff=True do?

    Delays grow exponentially up to retry_backoff_max, which gives the failing service time to recover.
  3. 3. Why add jitter to retries?

    Random spread prevents synchronized retry storms that keep an overloaded service down.
  4. 4. Which exceptions should normally be listed in autoretry_for?

    Retrying bugs wastes resources and hides the real problem. Retry only errors that may succeed later.
  5. 5. What happens when max_retries is exceeded and you called self.retry(exc=exc)?

    After the last allowed retry, the supplied exception is re-raised and the task state becomes FAILURE.

Summary

  • Use bind=True and raise self.retry(exc=exc, countdown=...), or declare autoretry_for=(...).
  • max_retries, default_retry_delay, retry_backoff, retry_backoff_max and retry_jitter define the policy.
  • Retry only transient errors; exponential backoff with jitter protects shared services.
  • A retry frees the worker slot, unlike time.sleep.
  • Make tasks idempotent and alert when retries run out.

Next: Acks late and idempotency