Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 36 ยท Tasks and Workers

Calling Tasks: delay, apply_async, countdown and eta

Learn every way to call a Celery task: delay, apply_async, countdown, eta, expires, queue and priority, apply for local runs, and how to test with eager mode.

Defining a task is half the story. The other half is deciding how and when to run it. Celery gives you three calling styles and a handful of options that control timing, routing and lifetime.

The three ways to call

Python
add(2, 2)                      # 1. direct call: runs here, now, no broker
add.delay(2, 2)                # 2. shortcut: send a message
add.apply_async((2, 2))        # 3. full form: send a message, with options

delay() is shorthand: add.delay(2, 2, debug=True) equals add.apply_async((2, 2), {"debug": True}). Use apply_async as soon as you need an option, because delay cannot take any. Note that options go after the args and kwargs, as keyword arguments to apply_async; they are not task arguments.

Python
add.apply_async((2, 2), countdown=10)                    # run in ten seconds
add.apply_async(args=(2, 2), kwargs={}, queue="math")    # choose the queue

There is also apply(), which runs the task locally and synchronously but wraps it in the usual task machinery (a request object, an EagerResult). It is handy in tests and shells.

Timing options

Option Meaning
countdown=30 Do not run until at least 30 seconds from now.
eta=datetime Do not run before this exact time (use a timezone-aware datetime).
expires=60 or a datetime Discard the message if it has not started by then.
time_limit, soft_time_limit Override run-time limits for this call.

Two important details:

  • countdown and eta are lower bounds. The worker holds the message in memory until the time arrives, then queues it; if the worker is busy it can start later still.
  • expires is the opposite: an upper bound on waiting. A task that has not started by then is revoked on arrival. It is perfect for work that becomes useless if late, such as a "your code is 482913" SMS.

The stand-in below keeps messages in a priority queue ordered by their ETA on a simulated clock, then applies expiry rules. No real waiting is involved.

Python Stand-in: countdown, eta and expires on a simulated clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You can see the three effects: resize_image is delayed only because the worker was busy; send_reminder waits for its ETA; the SMS tasks both expired because the worker did not start them in time.

Routing options

Python
add.apply_async((2, 2), queue="math")                   # send to a named queue
add.apply_async((2, 2), routing_key="math.add", exchange="tasks")  # AMQP style
add.apply_async((2, 2), priority=0)                     # 0 is highest on RabbitMQ

Priority works differently on different brokers. RabbitMQ supports it when queues are declared with x-max-priority; Redis emulates a few levels by using separate lists; SQS has no priorities at all. Queues and routing have their own lesson: Queues and routing.

Other useful options

Python
add.apply_async((2, 2), link=log_result.s())            # run on success
add.apply_async((2, 2), link_error=on_error.s())        # run on failure
add.apply_async((2, 2), serializer="json")              # per-call serializer
add.apply_async((2, 2), retry=True,
                retry_policy={"max_retries": 3, "interval_start": 0.2})
add.apply_async((2, 2), headers={"tenant": "acme"})     # extra message headers
add.apply_async((2, 2), task_id="my-own-id-123")        # choose the id yourself

The retry option here is about publishing the message (when the broker connection fails), not about re-running the task after errors. Mixing the two up is common.

A custom task_id is useful for de-duplication: if an id must be unique per business entity, such as invoice-1042, you know where to look the result up without storing the id elsewhere. Make sure ids really are unique, because a reused id overwrites an earlier result.

Running locally: apply() and eager mode

add.apply((2, 2)) runs immediately in your process and returns an EagerResult whose .get() returns at once. The global switch is task_always_eager = True, which makes every delay() and apply_async() behave like apply(). It is useful in unit tests:

Python
app.conf.task_always_eager = True
app.conf.task_eager_propagates = True     # raise exceptions instead of swallowing them

Eager mode has limits: it skips the broker, serialization and routing, so bugs related to those (an unserializable argument, a wrong queue name) do not appear. Treat it as a way to test your task logic, and add at least one integration test that goes through a real broker.

This stand-in mimics the difference between a queued call and an eager call, including the serialization step that eager mode skips:

Python Stand-in: eager mode skips serialization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The eager call hides a bug that a real queue would reveal immediately. That is why a green eager test suite does not prove your tasks will work in production.

Calling from inside a task

A task may call other tasks with delay(). Do not wait for them with .get() inside the task body; Celery raises a warning for good reason, because a worker waiting for a result occupies a slot that the awaited task may need. If you need "do B after A", use a workflow (chains, covered in a later chapter) or pass the next step as link=.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which call lets you set countdown=10?

    delay() passes every keyword to the task itself. Only apply_async accepts execution options.
  2. 2. What does expires=60 do?

    expires limits how long a message may wait. Run-time limits are time_limit and soft_time_limit.
  3. 3. A test passes with task_always_eager=True but the task fails in production with an encoding error. Why?

    Eager calls never serialize arguments, so unserializable values are not detected.
  4. 4. What is apply_async(..., retry=True) about?

    That option controls publishing retries only. Task retries use self.retry or autoretry_for.

Summary

  • delay() is the short form; apply_async() accepts options such as countdown, eta, expires, queue, priority, link and task_id.
  • countdown and eta set earliest start; expires sets the latest start.
  • apply() and task_always_eager run tasks locally, which is good for logic tests but skips the broker and serialization.
  • Never block on another task's result inside a task.

Next: Task Results, States and AsyncResult