Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 36 ยท Workflows: Chains and Groups

Celery Chords: Parallel Tasks With a Callback

Learn Celery chords: run a group in parallel then call one callback with all results. Covers the header, body, backend needs, error behavior and pitfalls.

A chord is a group plus a callback. The group (called the header) runs in parallel. When every header task has succeeded, one more task (the body) runs and receives the list of all results. This is the classic fan-out / fan-in or map-reduce shape: "download 10 pages in parallel, then merge them", "score every candidate, then pick the best".

Celery, brokers and workers do not run on this site. Celery code is read-only; the runnable examples are plain-Python stand-ins that count finished members the way a chord does.

Building a chord

Python
from celery import chord

@app.task
def word_count(text):
    return len(text.split())

@app.task
def total(counts):          # receives a LIST
    return sum(counts)

header = [word_count.s(t) for t in documents]
result = chord(header)(total.s())          # starts it, returns AsyncResult of body
result.get()                               # total words

chord(header)(body) sends the header group right away and arranges for body to be sent when the last header task finishes. The AsyncResult you get back belongs to the body. The body receives one argument: the list of header return values, in header order.

The longer, more explicit spelling is useful when you want options:

Python
workflow = chord(header, body=total.s().set(queue="reports"))
workflow.apply_async()

Chords made from chains

You rarely write chord(...) by hand in larger code. Celery builds one when a group is followed by another signature in a chain:

Python
workflow = group(word_count.s(t) for t in documents) | total.s()
workflow.delay()

Because the number of header tasks usually depends on an earlier result, the common pattern is a task that builds the chord at runtime:

Python
@app.task
def process_folder(folder):
    paths = list_files(folder)
    return chord(word_count.s(p) for p in paths)(total.s())

or use replace inside a task (self.replace(chord(...))) so the current task id is taken over by the new workflow. For fixed shapes the | form is perfectly fine: group(...) | total.s() creates a chord.

How a chord counts

When a header task finishes, it records its state in the result backend and increments a counter. The task that observes "all members done" sends the body. The consequences:

  • A chord requires a result backend that supports it. Redis, Memcached, Database and others do; a missing or rpc:// backend cannot support chords reliably.
  • Header results are stored until the body runs, so keep them small.
  • Header tasks must not set ignore_result=True; otherwise the counter has nothing to read.
Python Stand-in: a chord fires its body after the last member

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When a header task fails

If any header task fails (and is not retried to success), the body is never called. Instead the body's AsyncResult goes to FAILURE with a ChordError that names the failed member. Any link_error on the chord body fires.

Python
result = chord(header)(total.s().on_error(report_failure.s()))

Plan for it: make header tasks retry transient errors (Retries and backoff), and let them return a sentinel (for example None or a dict with an error key) for failures you can tolerate, so the body can decide what to do with a partial list.

Python Stand-in: tolerate partial failures in the body

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A chord inside a chain

A chord is a signature, so you can place it in the middle of a chain:

Python
pipeline = (
    prepare.s(job_id)
    | chord([work.s(i) for i in range(5)], merge.s())
    | publish.s()
)

merge gets the list of five results; publish gets merge's return value.

Chord pitfalls

  1. No backend, or the wrong one. Symptoms: the body never runs, or you see "chord_unlock" tasks retrying. Use Redis or a database backend.
  2. Huge result lists. The body receives all results in memory. Return ids or file paths, not documents.
  3. An empty header. With zero members the body runs with an empty list in recent versions; check your version and guard in the body anyway.
  4. Waiting inside tasks. Never block on result.get() in a task; a chord removes that need.
  5. Hidden retries. If a header task is retried, the counter only moves when it finally succeeds, so the body waits for it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the chord body receive as input?

    The body gets a single argument: the list of return values in the order the header tasks were defined.
  2. 2. When does the chord body run?

    The body waits for all header tasks to finish successfully.
  3. 3. What happens to the body if one header task fails for good?

    A failed header member prevents the body; the body's state is FAILURE and error callbacks fire.
  4. 4. Which setup breaks chords?

    Chords count finished members in the backend, so it must exist and header results must be stored.

Summary

  • A chord = header group in parallel + one body task that gets the list of results.
  • chord(header)(body) or group(...) | body.s() creates it; the returned result belongs to the body.
  • It needs a real result backend and stored header results.
  • A permanently failing header task blocks the body; design tasks to retry or to return tolerable error values.
  • Keep results small and never wait on results inside tasks.

Next: map, starmap and chunks