Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 36 ยท Workflows: Chains and Groups

Celery Groups: Run Tasks in Parallel

Use Celery groups to run many tasks in parallel, collect results with GroupResult, handle partial failures, and combine groups with chains.

A group sends several tasks at once and lets any available workers run them in parallel. Where a chain is a pipe, a group is a fan-out: "resize these 20 images", "ask 5 price APIs", "send this notice to every subscriber in the batch". The group finishes when every member has finished.

Celery and its workers do not run on this site, so Celery code is read-only. The runnable examples are plain-Python stand-ins (a thread-free simulation of workers) that follow the same result and failure semantics.

Creating a group

Python
from celery import group

@app.task
def resize(path, width):
    return make_thumbnail(path, width)

jobs = group(resize.s(p, 200) for p in ["a.jpg", "b.jpg", "c.jpg"])
result = jobs.apply_async()          # returns a GroupResult

A generator expression works, as does a list. jobs.apply_async() publishes all three messages straight away. Idle worker processes pick them up in any order; with enough workers they run simultaneously.

The returned GroupResult is a collection of the member AsyncResult objects plus helpers:

Python
result.ready()          # True when ALL members are finished
result.successful()     # True when ALL members succeeded
result.failed()         # True when ANY member failed
result.completed_count()  # how many are done (successfully)
result.get(timeout=60)  # list of return values, in the order you defined them
result.join()           # same as get(), older name

The order of get() follows the definition order, not the finishing order. So ["a.jpg", "b.jpg", "c.jpg"] always yields results in that order even if c.jpg finished first.

Groups share one set of arguments or many

group takes any iterable of signatures, so each member can differ:

Python
group(
    fetch_price.s("shop-a", sku),
    fetch_price.s("shop-b", sku),
    check_stock.s("warehouse", sku),     # a different task entirely
)

You can also set options per member: fetch_price.s(...).set(queue="io").

Python Stand-in: a group collects results in order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Partial failure

If one member fails, the others still run: members are independent. But result.get() raises the first member's exception it meets. To collect what succeeded, use get(propagate=False) and inspect each value, or walk the results yourself:

Python
result = jobs.apply_async()
values = result.get(propagate=False)       # exceptions come back as values
for res in result.results:
    if res.failed():
        log.warning("task %s failed: %s", res.id, res.result)

For per-task resilience configure retries on the member task, so a transient network blip does not make the whole group fail.

Python Stand-in: keep the good results, report the bad

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Groups need a result backend

To read results (get, ready, successful) Celery needs a result backend. A group also saves its own metadata (the list of member ids) with GroupResult.save() if you want to restore it later in another process:

Python
result = jobs.apply_async()
result.save()
restored = app.GroupResult.restore(result.id)

If you never read results ("fire and forget"), set ignore_result=True on the tasks and skip the backend cost.

Groups inside chains

A group can be one step of a chain. The next step then needs all the group's results, which is what a chord does. Celery turns chain(group, task) into a chord automatically:

Python
workflow = chain(
    split_job.s(job_id),                     # returns a list of part ids
    group(process_part.s(i) for i in range(4)),
    merge_parts.s(),                         # gets the list of 4 results
)

See Chords for the exact rules, including the backend requirement.

Tips

  • Do not create giant groups of 100,000 tiny tasks; message overhead dominates. Use chunks to batch.
  • Keep members independent: they may run in any order or on different machines.
  • group.skew(start=1, stop=10, step=1) spreads countdowns so a burst of tasks does not hit an external API at the same instant.
  • Do not call result.get() inside another task; use a chord or callback.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In what order does GroupResult.get() return values?

    get() returns results in definition order, independent of which task finished first.
  2. 2. One task in a group raises an exception. What happens to the other tasks?

    A group has no all-or-nothing behavior. get() will raise, but the other members run normally.
  3. 3. How do you collect exceptions as values instead of raising?

    propagate=False returns exceptions in the result list instead of re-raising the first.
  4. 4. What does chain(group(...), task.s()) become automatically?

    The next step must wait for every group member, which is exactly a chord.
  5. 5. Why avoid a group of 100,000 tiny tasks?

    Each task is a separate message, so very small tasks waste more time in messaging than in work.

Summary

  • group(sig1, sig2, ...) publishes all tasks at once; workers run them in parallel.
  • GroupResult offers ready, successful, failed, completed_count and get, which returns values in definition order.
  • Members are independent; use get(propagate=False) for partial results.
  • A group in a chain becomes a chord; batch tiny jobs with chunks.
  • Reading results requires a result backend.

Next: Chords: parallel work plus a callback