Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 36 ยท Workflows: Chains and Groups

Celery map, starmap and chunks: Batch Many Items

Learn the Celery canvas helpers map, starmap and chunks to process large lists efficiently, choose batch sizes, and combine them with groups and chords.

A group creates one message per task. For ten tasks that is perfect; for a million rows it is wasteful, because sending and receiving a message costs more than doing a tiny piece of work. Celery therefore offers three helpers that process a whole list inside one task message: map, starmap and chunks. This lesson shows what each does and when to use it.

Celery does not run on this site. The Celery code is read-only; the runnable examples are plain-Python stand-ins that mimic how items are split and applied.

map: one task, many single arguments

task.map(iterable) applies the task to every item, sequentially inside one worker task:

Python
@app.task
def square(n):
    return n * n

square.map([1, 2, 3, 4]).delay()     # one message; worker returns [1, 4, 9, 16]

It behaves like [square(n) for n in items] executed by a single worker process. The result is the list of return values. Nothing runs in parallel, so use it to cut message overhead, not to speed up heavy work.

starmap: one task, many argument tuples

starmap is map for tasks with several arguments. Each item is a tuple that is unpacked into the call:

Python
@app.task
def add(x, y):
    return x + y

add.starmap([(2, 2), (4, 4), (8, 8)]).delay()    # [4, 8, 16]

The name comes from the * in add(*item), and the idea matches Python's itertools.starmap.

Python Stand-in: map and starmap in one process

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

chunks: split a long list into batches

chunks(iterable, n) splits the list into pieces of n items and creates one task per piece. Each piece is processed like a starmap. Unlike map, the pieces can run on different workers at once:

Python
res = add.chunks(zip(range(100), range(100)), 10)   # 10 tasks of 10 additions
group_result = res.group().apply_async()

chunks(...) itself is a signature, not a group. Call .group() to turn it into a group that can be sent. The result is a list of lists: one inner list of values per chunk.

You choose n to balance two costs:

Chunk size Effect
Very small Many messages, high overhead, good parallelism
Very large Few messages, but one slow chunk delays everything; poor load balancing
Moderate (tens to a few thousand items) Usually the best balance; measure it
Python Stand-in: split work into chunks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Combining with groups and chords

Because chunks returns a signature, you can make results usable downstream. A common pattern is a chord whose body flattens and reduces:

Python
from celery import chord

@app.task
def process_batch(rows):            # a normal task that takes a list
    return [transform(r) for r in rows]

@app.task
def combine(batches):
    return [item for batch in batches for item in batch]

batches = [rows[i:i + 500] for i in range(0, len(rows), 500)]
chord(process_batch.s(b) for b in batches)(combine.s())

Writing your own batch task (one that accepts a list) is often clearer than chunks: it lets you share setup work like opening a connection once per batch. Use chunks when a ready-made single-item task should just be reused.

Choosing the right tool

Need Use
A few independent jobs in parallel group
Reduce message overhead, no parallelism needed map / starmap
Large list, parallel across workers, reusing a single-item task chunks
Large list, shared setup per batch custom batch task + group or chord
Wait for all, then combine chord

Pitfalls

  1. Expecting map to be parallel. It is not; it is one task.
  2. Huge arguments in the message. Chunking a list of large objects puts all of them into message bodies. Pass ids and load data in the worker.
  3. Time limits. A map or chunk is a single task, so its whole running time counts against the task's time limit. Large chunks can hit the hard limit.
  4. Partial failure. If one item in a map raises, the whole task fails and the remaining items in that task do not run. Catch per-item errors inside your own function if you need to continue.
  5. Result size. A million-item result list is a large backend value. Write results to storage and return a count.
Python Stand-in: one bad item fails a whole map task

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How many task messages does add.starmap([(1, 2), (3, 4), (5, 6)]).delay() send?

    map and starmap run the whole list inside a single task message, one item after another.
  2. 2. What is the difference between map and chunks?

    chunks splits the list into batches and each batch becomes its own task; map keeps everything in one task.
  3. 3. What does a starmap item look like for add(x, y)?

    starmap unpacks each tuple with *, so (2, 3) becomes add(2, 3).
  4. 4. How do you send a chunks signature?

    chunks(...) is a signature; .group() turns it into a group you can send.
  5. 5. Why can a very large chunk be risky?

    A big chunk runs a long time as a single task, risking the time limit and delaying completion.

Summary

  • map and starmap apply a task to many items inside one task message, sequentially.
  • chunks(items, n) makes one task per batch, which can run in parallel; call .group() to send it.
  • Pick the batch size by measuring: small enough to balance load, large enough to amortize message cost.
  • One failing item fails the whole map task or chunk; handle per-item errors yourself if needed.
  • For shared setup per batch, a custom batch task with a group or chord is often clearer.

Next: Retries and exponential backoff