Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 36 ยท Workflows: Chains and Groups

Celery Chains: Run Tasks in Sequence

Build Celery chains with chain() and the pipe operator, understand how results flow between tasks, handle failures, and read results from the last task.

A chain runs tasks one after another, feeding each task's return value into the next. It is the Celery version of a Unix pipe: download | resize | upload. Because the whole chain is described up front, the workers can pick up each step as soon as the previous one finishes, and your web request never waits for any of them.

Celery and a broker do not run on this site. The Celery code blocks are read-only, and the runnable examples are plain-Python stand-ins that reproduce how arguments and errors travel through a chain.

Your first chain

Python
from celery import chain

@app.task
def fetch(url):
    return download(url)          # returns bytes or a path

@app.task
def parse(content):
    return extract_items(content)

@app.task
def store(items):
    return save_all(items)

workflow = chain(fetch.s("https://example.com/feed"), parse.s(), store.s())
result = workflow.apply_async()

parse.s() has no arguments of its own. When fetch finishes, its return value is prepended to parse's arguments (the rule from Signatures), so the worker runs parse(fetch_result). The same happens between parse and store.

apply_async() sends only the first task. When it succeeds, the worker reads the rest of the chain from the message headers and publishes the next task. That means each step can run on a different worker, in a different queue, minutes apart.

The pipe operator

The | operator builds the same chain and reads naturally:

Python
workflow = fetch.s(url) | parse.s() | store.s()
workflow.delay()

Chains are signatures themselves, so you can chain chains, put them inside groups, or store them in a variable and send them later.

What you get back

workflow.delay() returns an AsyncResult for the last task in the chain. Its parent attribute points to the one before, and so on:

Python
result = workflow.delay()
result.get(timeout=30)        # waits for store(), returns its value
result.parent                 # AsyncResult of parse()
result.parent.parent          # AsyncResult of fetch()
result.parent.parent.parent   # None: the start of the chain

Reading results needs a result backend configured; without one get() raises an error. Avoid calling get() inside a task, because a worker that waits for another task can deadlock when all workers are busy waiting. Prefer callbacks or further chain steps.

Immutable steps

If a step does not want the previous result, use .si():

Python
chain(
    build_report.s(report_id),
    upload.s(),                     # receives the report path
    notify_team.si("report ready"), # ignores upload's return value
)

Without .si() the notification task would receive an unexpected extra argument and raise TypeError.

Partial arguments in the middle

Mix fixed and passed values freely. Remember the previous result comes first:

Python
chain(add.s(1, 2), add.s(10), add.s(100))
# add(1, 2) -> 3, then add(3, 10) -> 13, then add(13, 100) -> 113
Python Stand-in: a chain passes the result forward

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What happens when a step fails

If any task in a chain raises an exception (and is not retried), the chain stops. Tasks after the failing one are never sent. The AsyncResult of the failing step has state FAILURE, and calling get() on the last result re-raises the original exception, because the failure is propagated to its dependants.

Python
result = (fetch.s(url) | parse.s() | store.s()).delay()
try:
    result.get(timeout=60)
except Exception as exc:
    print("chain failed:", exc)

To react to failure without polling, attach an error callback with link_error. See Failure handling. For temporary problems, give each task its own retry policy so one flaky download does not kill the whole pipeline.

Python Stand-in: a failing step stops the chain

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Design tips

  • Keep payloads small. The return value of each task travels through the broker/backend. Pass ids, file paths or object keys, not 50 MB blobs.
  • Make steps idempotent (Acks late and idempotency), because a step can be delivered twice.
  • Use different queues per step when workloads differ: fetch.s(url).set(queue="io") | resize.s().set(queue="cpu"). See Queues and routing.
  • Prefer chains over tasks that call .delay() for the next step. A chain is visible as one workflow, supports parent lookups and error propagation, and the next step is not lost if the calling task dies.
  • Group steps in the middle when one stage fans out; see Groups. A group inside a chain is automatically upgraded to a chord.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In chain(add.s(1, 2), add.s(10)), what does the second task run?

    The first task returns 3, which is prepended to the second signature's arguments, giving add(3, 10).
  2. 2. What does chain.apply_async() return?

    You get the AsyncResult of the last task. Its parent attribute leads back through the chain.
  3. 3. A task in the middle of a chain raises an unhandled exception. What happens to the later tasks?

    The chain stops at the failing task, and the error is propagated to the results of the dependent tasks.
  4. 4. How do you stop a task in a chain from receiving the previous result?

    Immutable signatures ignore the parent result, so they only run after the previous step has finished.

Summary

  • chain(a.s(), b.s(), c.s()) or a.s() | b.s() | c.s() runs tasks in sequence, passing each result as the first argument of the next.
  • Only the first task is sent at once; each worker publishes the next step on success.
  • The returned AsyncResult belongs to the last task; .parent walks backwards.
  • A failure stops the chain and propagates to dependants; use retries and error callbacks for resilience.
  • Use .si() for steps that need no input, and keep passed data small.

Next: Groups: run tasks in parallel