Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 36 ยท Tasks and Workers

Task Design Rules: Idempotent, Small and Safe

Design Celery tasks that survive retries and duplicates: idempotency, passing ids instead of objects, small arguments, safe serialization, acks_late and time limits.

Celery makes it easy to move a function into the background. It does not make it safe. A task runs in a different process, possibly twice, possibly long after it was queued, and possibly on a database that has changed in the meantime. Good task design is mostly about accepting those facts.

Rule 1: assume the task can run more than once

Message systems usually promise at-least-once delivery. Reasons for a repeat:

  • The worker crashed after doing the work but before acknowledging (acks_late=True).
  • The broker did not receive the ack and redelivered the message (Redis does this after the visibility timeout).
  • You retried after a timeout even though the first attempt actually succeeded.
  • A user double-clicked a button and your code enqueued twice.

So write tasks to be idempotent: running them twice gives the same final state as running them once.

  • Bad: balance += 10 as a "task". Twice means +20.
  • Good: "set the invoice's status to paid and record payment pay_8841, unless that payment is already recorded".

Common techniques are natural keys with unique constraints, get_or_create/INSERT ... ON CONFLICT DO NOTHING, a "processed" flag checked inside a transaction, and idempotency keys passed by the caller. This runnable stand-in compares a naive task with an idempotent one under redelivery:

Python Stand-in: redelivery and idempotency

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a real system the set payments_done is a database table with a unique constraint, and the check and the update share one transaction; otherwise two workers could both pass the check at the same moment.

Rule 2: pass identifiers, not objects

Python
# Bad: an ORM object, a datetime, a big dict
send_invoice.delay(invoice)

# Good: ids and simple values
send_invoice.delay(invoice.id)

Reasons:

  1. Serialization. Messages are JSON by default. Model instances, datetime, Decimal, set and UUID are not JSON types. Using pickle to get around this is a security risk (see below).
  2. Staleness. An object serialized now describes the database as it was. By the time the worker starts, the invoice may have changed. Loading by id inside the task gives you current data.
  3. Size. Big messages slow the broker. Keep arguments small (a few hundred bytes); pass a reference to larger data.

Inside the task, re-fetch and re-check:

Python
@app.task
def send_invoice(invoice_id):
    invoice = Invoice.objects.filter(pk=invoice_id).first()
    if invoice is None or invoice.sent_at:
        return                      # deleted or already sent: nothing to do
    ...

Rule 3: enqueue after the transaction commits

A classic race in web apps: you create a row, call task.delay(row.id) inside the same transaction, and the worker runs before the commit. It looks for the row and does not find it. Django offers transaction.on_commit:

Python
from django.db import transaction

with transaction.atomic():
    order = Order.objects.create(...)
    transaction.on_commit(lambda: process_order.delay(order.id))

The same idea applies to other frameworks: enqueue only after the data you reference is visible. The stand-in shows the race in miniature, with a "database" that becomes visible only at commit:

Python Stand-in: the enqueue-before-commit race

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rule 4: use safe serialization

Celery supports several serializers: json (default since Celery 4), pickle, yaml, msgpack. Avoid pickle for anything that crosses a trust boundary. Unpickling can execute arbitrary code, so anyone who can write to your broker could run code on every worker. Keep the default:

Python
app.conf.task_serializer = "json"
app.conf.result_serializer = "json"
app.conf.accept_content = ["json"]       # workers refuse other formats

Only accept pickle if the broker is private and fully trusted, and you have a real need to ship Python objects. For richer data, convert at the edges: ISO strings for datetimes, strings for Decimal, str(uuid) for UUIDs.

Rule 5: set time limits

A task that hangs holds a worker slot forever. Set limits globally and override where needed:

Python
app.conf.task_soft_time_limit = 300     # raises SoftTimeLimitExceeded in the task
app.conf.task_time_limit = 360          # hard: the child process is killed

The soft limit lets you clean up:

Python
from celery.exceptions import SoftTimeLimitExceeded

@app.task(soft_time_limit=60, time_limit=90)
def export_all(user_id):
    try:
        ...
    except SoftTimeLimitExceeded:
        mark_export_failed(user_id)

Time limits work with the prefork pool. Also set timeouts in the libraries the task uses (HTTP clients, database drivers); a socket without a timeout can hang past what you expect.

Rule 6: choose when to acknowledge

Setting Behaviour Trade-off
acks_late=False (default) Ack just before running A crash loses the task; it never runs twice.
acks_late=True Ack after the task finishes A crash redelivers the task; it may run twice.

Use acks_late=True for important tasks, but only together with idempotency. Pair it with task_reject_on_worker_lost=True if you want tasks killed by an out-of-memory event to be requeued rather than dropped.

Rule 7: keep tasks small and focused

Long monolithic tasks are hard to retry: failing at step 9 of 10 repeats everything. Prefer several small tasks connected by workflows, each idempotent on its own. To process 100,000 rows, queue one task per chunk of a few hundred rows instead of one giant task or 100,000 tiny ones.

Also never share state through globals between runs, do not rely on request-specific variables (current user, session) inside tasks, and log with the task id so you can trace one run through the system.

A short checklist

  • Arguments: ids and plain values, JSON-serializable, small.
  • Body: reload data, check preconditions, make the effect idempotent.
  • Safety: time limits, acks_late if loss is worse than duplication.
  • Timing: enqueue after commit; no blocking .get() inside tasks.
  • Observability: a log line with the task id at start and end.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should tasks be idempotent?

    Crashes, redelivery and retries can run the same message again. Idempotent tasks make that harmless.
  2. 2. What is the best argument for a task that emails an invoice?

    An id is small, serializable and lets the worker load fresh data.
  3. 3. Why use transaction.on_commit before calling delay() for a newly created row?

    If the task starts before the commit, the worker may not find the row.
  4. 4. What is the main risk of the pickle serializer?

    Anyone able to publish messages could run code on your workers, so prefer JSON and accept_content=['json'].
  5. 5. What does acks_late=True change?

    Late ack trades possible duplicates for protection against lost tasks.

Summary

  • Expect duplicates: make tasks idempotent with keys, unique constraints and precondition checks.
  • Pass ids and simple JSON values; reload data inside the task.
  • Enqueue after commit, keep JSON as the serializer, set soft and hard time limits.
  • Pick acks_late consciously, and split big jobs into small, retryable tasks.

Next: Message Brokers Compared: Redis, RabbitMQ and SQS