Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 40 ยท Tasks and Gathering

asyncio.gather: Running Many Coroutines at Once

Learn asyncio.gather to run many awaitables concurrently, collect results in order, handle errors with return_exceptions, and gather a dynamic list of jobs.

When you have several independent jobs and you want all their results, asyncio.gather is the shortest path. You hand it a list of awaitables, it runs them concurrently, and it returns their results in a list in the same order you passed them, regardless of which finished first.

The basics

gather(*awaitables) wraps each coroutine in a task for you, waits until every one has finished, and returns the results as a list:

Python gather returns results in order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

pear finishes first, but its result still sits in the middle of the list. Order of results equals order of arguments, which makes it easy to unpack: apple, pear, plum = await asyncio.gather(...). The whole call takes as long as the slowest job (0.3 seconds), not the sum.

Gathering a dynamic list

Usually you do not know the number of jobs in advance. Build a list of coroutines and unpack it with *:

Python Gathering a list of jobs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

*(generator) expands the generator into separate arguments. Do not write await asyncio.gather([...]): gather takes awaitables as separate arguments, not a single list.

What happens when one job fails

By default the first exception raised by any job is thrown from await gather(...) right away. The other jobs are not cancelled; they keep running in the background, and their results are lost. This surprises many people:

Python Default error behavior

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

gather raised at 0.05 seconds, yet A and B still finished later. If you did not wait, their output would be lost or they would be cancelled when the program ends. For all-or-nothing work where one failure should stop the rest, prefer a TaskGroup.

return_exceptions=True: collect everything

When you want every outcome, successes and failures alike, pass return_exceptions=True. Exceptions are returned in the result list instead of being raised:

Python return_exceptions=True

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is the standard "best effort" pattern for batch jobs: process everything, then report which items failed. Always check each result with isinstance(result, BaseException).

With return_exceptions=True, a CancelledError of an inner task is also returned as a value rather than raised. Cancellation of the gather call itself still propagates normally.

Cancelling a gather

If the task that is awaiting gather is cancelled, gather cancels all its children. You can also keep a handle on the group: group = asyncio.gather(...), then group.cancel() cancels every child. Without return_exceptions, one child raising does not cancel the siblings, as seen above.

gather versus a loop of awaits

A list comprehension with await is sequential. These two lines look similar but behave very differently:

Python
results = [await fetch(u) for u in urls]            # one at a time
results = await asyncio.gather(*(fetch(u) for u in urls))   # all at once

The second line starts all the requests together. For hundreds of jobs you usually also want to limit how many run at the same time, using a semaphore.

Gather with tasks you already created

gather accepts tasks and futures as well as coroutines, so you can create tasks first, do other work, then gather them. Results still arrive in argument order:

Python Gathering existing tasks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In what order does gather return its results?

    Results line up with the arguments, no matter which job finishes first.
  2. 2. A job in gather raises an exception and return_exceptions is False. What happens to the other jobs?

    The first exception propagates immediately; siblings are not cancelled by default, which is why TaskGroup is often safer.
  3. 3. What does return_exceptions=True change?

    You get one entry per job, either a value or an exception object.
  4. 4. Which line correctly gathers jobs built from a list?

    gather takes awaitables as separate positional arguments, so unpack the list with *.

Summary

  • await asyncio.gather(a, b, c) runs them concurrently and returns results in argument order.
  • Build dynamic batches with gather(*(coro(x) for x in items)).
  • By default the first exception is raised and siblings keep running; return_exceptions=True collects every outcome.
  • Cancelling the awaiting task cancels all children.
  • Combine with a semaphore to limit concurrency.

Next: TaskGroups and structured concurrency