Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท Retries and Reliability

Celery Failure Handling, Callbacks and Dead Letters

Handle Celery task failures: link and link_error callbacks, Task.on_failure and on_retry hooks, signals, dead-letter patterns and alerting for exhausted retries.

Tasks fail. Sometimes for a good reason (bad input, a vendor outage), sometimes because of a bug. A reliable system does not only retry; it also makes sure failures are seen, recorded and recoverable. Celery offers four layers for that: callbacks on a signature, hooks on the task class, global signals, and result states. This lesson shows each one and ends with a dead-letter pattern for jobs that cannot succeed.

Celery does not run on this site. Celery code is read-only; the runnable examples are plain-Python stand-ins that reproduce the hook order and the dead-letter logic.

Task states you will meet

PENDING (unknown or not yet received) โ†’ STARTED (only if task_track_started is enabled) โ†’ SUCCESS, FAILURE, RETRY or REVOKED. A failed task stores its exception and traceback in the result backend (result.traceback). result.state, result.failed() and result.get(propagate=False) let you inspect it.

A signature can name follow-up tasks for success and for failure:

Python
send_receipt.apply_async(
    args=[order_id],
    link=log_success.s(),                 # runs after success, receives the result
    link_error=alert_on_failure.s(order_id),
)

The error callback receives the task id of the failed task (not the exception). Look the exception up from the backend if you need it:

Python
@app.task
def alert_on_failure(failed_task_id, order_id=None):
    result = app.AsyncResult(failed_task_id)
    notify_ops(f"task {failed_task_id} failed for order {order_id}: {result.result!r}")

Newer Celery versions can also call an errback that accepts (request, exc, traceback); the task-id form above is the classic one. The on_error() helper on a signature is shorthand for link_error. In a chain or chord you can attach the errback to the whole workflow so one handler covers every step.

Hooks on a custom Task class

For behavior that every task should share, subclass Task:

Python
from celery import Task

class AuditedTask(Task):
    def on_success(self, retval, task_id, args, kwargs):
        audit_log.info("ok %s", task_id)

    def on_retry(self, exc, task_id, args, kwargs, einfo):
        audit_log.warning("retrying %s: %s", task_id, exc)

    def on_failure(self, exc, task_id, args, kwargs, einfo):
        audit_log.error("failed %s: %s", task_id, exc)
        errors.capture(exc)                      # send to Sentry or similar

@app.task(base=AuditedTask, bind=True, autoretry_for=(IOError,), max_retries=3)
def export_file(self, path):
    ...

on_failure runs in the worker after the final failure (not for each retry; on_retry handles those). Keep hooks tiny and exception-safe: an error inside a hook can hide the original failure.

Signals

Signals are global hooks that need no changes to tasks: task_failure, task_retry, task_success, task_prerun, task_postrun, task_revoked, worker_ready and others.

Python
from celery.signals import task_failure

@task_failure.connect
def report_failure(sender=None, task_id=None, exception=None, args=None, kwargs=None, **extra):
    metrics.increment("celery.task.failed", tags=[f"task:{sender.name}"])

Signals are the best place for cross-cutting concerns: metrics, error tracking, logging correlation ids.

Python Stand-in: hook order for success, retry and failure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Dead-letter pattern

A dead-letter store holds jobs that exhausted their retries, so a human or a script can inspect and replay them instead of the failure vanishing into a log. Celery has no built-in dead-letter table; build one in an on_failure hook or errback:

Python
class DeadLetterTask(Task):
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        DeadLetter.objects.create(
            task_name=self.name,
            task_id=task_id,
            args=args,
            kwargs=kwargs,
            error=repr(exc),
            traceback=str(einfo),
        )

Replaying is then app.send_task(letter.task_name, args=letter.args, kwargs=letter.kwargs) after you fixed the cause. RabbitMQ users can additionally use broker-level dead-letter exchanges (x-dead-letter-exchange) for rejected messages; that is a separate, broker feature.

Python Stand-in: a dead-letter list with replay

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What to alert on

  • Failure rate per task name, from task_failure signals.
  • Exhausted retries: count dead letters; alert when the number grows.
  • Queue age: the oldest message waiting (see Inspecting workers).
  • Unexpected exceptions (bugs): send to an error tracker with the task name and arguments (scrubbed of secrets).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does link_error attach to a signature?

    link_error names error callbacks, while link names success callbacks.
  2. 2. When does Task.on_failure run?

    on_retry handles each retry; on_failure runs for the final failure.
  3. 3. Which is best for adding metrics to every task without editing them?

    Signals are global hooks, ideal for cross-cutting concerns like metrics and error tracking.
  4. 4. What is a dead-letter store used for?

    It preserves failed jobs with their arguments and errors, so nothing is silently lost.

Summary

  • Failed tasks record state and traceback in the result backend.
  • link and link_error attach success and failure callbacks to a signature or a whole workflow.
  • A custom Task base class offers on_success, on_retry and on_failure; signals add global hooks.
  • Build a dead-letter table for jobs that cannot succeed, and replay them after fixing the cause.
  • Alert on failure rate, exhausted retries and queue age.

Next: Inspecting workers and Flower