Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 36 ยท Retries and Reliability

Celery acks_late and Idempotent Tasks

Understand Celery message acknowledgement, acks_late, reject_on_worker_lost and visibility timeouts, and design idempotent tasks that are safe to run twice.

The most important sentence in this chapter: Celery delivers tasks at least once, not exactly once. A task may run twice, and your code must tolerate that. This lesson explains why, shows the settings that control the trade-off (acks_late, reject_on_worker_lost), and teaches the technique that makes duplicates harmless: idempotency.

Celery, brokers and workers do not run on this site. Celery code is read-only; the runnable examples are plain-Python stand-ins for a queue, a worker crash and idempotency keys.

Acknowledgements in 60 seconds

When a worker takes a message from the broker, the broker keeps the message as "unacknowledged" until the worker sends an ack. If the worker dies before acking, the broker puts the message back in the queue and another worker receives it.

Celery's default is early acknowledgement (acks_late=False): the worker acks the message just before running the task.

Setting Ack moment If the worker crashes mid-task Risk
acks_late=False (default) before running task is lost at-most-once
acks_late=True after finishing task is redelivered at-least-once (may run twice)

So early ack can lose work; late ack can duplicate work. For anything that must not be lost (orders, emails, payments) choose late ack and make the task idempotent.

Python
@app.task(acks_late=True, reject_on_worker_lost=True)
def charge_order(order_id):
    ...

Or globally in settings: task_acks_late = True.

reject_on_worker_lost=True handles the case where the worker process is killed (out of memory, kill -9): without it a late-ack task whose child process died is acked anyway on some setups and lost. With it, the message is rejected and requeued.

Visibility timeout (Redis and SQS)

Redis and SQS are not real message brokers with built-in acks. Celery simulates them with a visibility timeout (default one hour for Redis): if a message is not acked within that time, it is redelivered. Consequence: a task that legitimately runs longer than the timeout, or an ETA/countdown task further in the future than the timeout, can be executed twice at once. Raise the timeout above your longest task:

Python
app.conf.broker_transport_options = {"visibility_timeout": 43200}  # 12 hours

RabbitMQ does not have this problem (it uses real consumer acks and requeues on connection loss).

Idempotency: make duplicates harmless

An operation is idempotent if doing it twice has the same effect as doing it once. SET balance = 100 is idempotent; balance = balance + 10 is not.

Techniques:

  1. Set, don't increment. Store the final state, not a delta.
  2. Natural unique keys. Insert with a unique constraint and ignore duplicates (INSERT ... ON CONFLICT DO NOTHING).
  3. Idempotency key table. Before the side effect, record (task_name, business_key) with a unique constraint; if the row exists, return early.
  4. State machines. UPDATE orders SET status='paid' WHERE id=? AND status='pending'; the second run affects 0 rows and does nothing.
  5. Pass the key to external APIs. Payment and email providers accept an idempotency key so they deduplicate for you.
Python Stand-in: a crash causes redelivery

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the early-ack run the crash happened after the ack, so the message is gone and the second charge never happens (here the side effect had already happened, but for a crash before the side effect the job would be lost). In the late-ack run the message returns and the order is charged twice. The next example fixes the duplicate with an idempotency key.

Python Stand-in: an idempotency key prevents double charges

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a real system the check and the effect must be atomic: put both in one database transaction, or use the unique constraint as the check. A set in memory only works inside one process; workers are many processes.

Choosing a policy

  • Cheap, safe to repeat (cache warming, sending a "touch" request): default settings are fine.
  • Must not be lost (orders, emails): acks_late=True, reject_on_worker_lost=True, idempotent task.
  • Must not be repeated and may be lost (rare; some notifications): default early ack, and accept occasional loss.
  • Truly exactly-once is not offered by Celery; you build it with idempotency on top of at-least-once.

Also remember that retries intentionally run a task again, so idempotency protects you there too.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. With the default acks_late=False, what happens if a worker is killed in the middle of a task?

    Early acknowledgement removes the message before the work starts, so a crash loses the task.
  2. 2. What is the downside of acks_late=True?

    Because the message is acked after finishing, a crash before the ack leads to redelivery and a duplicate run.
  3. 3. Which operation is idempotent?

    The guarded update changes the row only once; a repeat finds nothing to change.
  4. 4. What does a too-short Redis visibility_timeout cause?

    If a message is not acked within the visibility timeout, Redis redelivers it, even though the first worker is still running it.

Summary

  • Celery is at-least-once with acks_late=True and at-most-once with the default early ack.
  • Use acks_late plus reject_on_worker_lost for work that must not be lost.
  • With Redis or SQS, set visibility_timeout above your longest task or ETA.
  • Make tasks idempotent with unique keys, guarded state changes and provider idempotency keys, with check and effect atomic.
  • Exactly-once is built on top, not provided.

Next: Time limits and revoking tasks