Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 40 ยท Async Libraries and Patterns

Async Iteration and Async Generator Patterns

Master async for, async generators, async comprehensions, aclosing, batching and merging streams of results, with runnable asyncio examples and common pitfalls.

Plenty of real data does not arrive all at once: lines from a socket, rows from a database cursor, pages from an API, messages from a queue. Python's answer is asynchronous iteration: a loop that can await between items. With async for and async generators you write code that reads like a normal generator pipeline while the event loop stays free to run other tasks between items.

This lesson is the toolbox for working with such streams: producing them, transforming them, merging them, stopping them safely, and avoiding the usual traps.

async for and async generators

An async generator is a function with both async def and yield. Calling it returns an async iterator, which you consume with async for. Inside the generator you may await anything.

Python Your first async generator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Async comprehensions ([... async for ...], also for sets, dicts and generator expressions) work inside any async def, or at the top level of the lesson editor. Remember that the loop awaits between items, so other tasks get to run in the gaps.

The async iterator protocol

async for works with any object that has __aiter__() returning an iterator, and an __anext__() coroutine that returns the next item or raises StopAsyncIteration. A class-based iterator is useful when you need state or cleanup that a generator does not express well:

Python A class-based async iterator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The built-ins aiter() and anext() (Python 3.10 and later) mirror iter() and next().

Transforming streams: pipelines

Because async generators can consume other async iterators, you can stack them like Unix pipes. Each stage handles one item at a time, so memory stays flat no matter how long the stream is.

Python A three-stage pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

batched is the pattern you use before bulk database inserts or API calls that accept lists. Python 3.12 added itertools.batched for ordinary iterables, but nothing for async ones, so a small helper like this is common.

flowchart LR
    A["source"] --> B["squares"] --> C["only_odd"] --> D["batched"] --> E["async for"]

Items flow one at a time from left to right; each stage awaits its upstream.

Closing generators safely

An async generator holds resources between yields (a connection, a file). If the consumer stops early with break or an exception, the generator is not finished: it is left suspended until the garbage collector or the event loop's shutdown hook closes it, and its finally blocks run at an unpredictable time. To make cleanup deterministic, wrap the generator in contextlib.aclosing():

Python aclosing makes cleanup predictable

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without aclosing, the "close cursor" line could print much later, after your program has moved on. Use aclosing() whenever you may stop consuming before the generator ends, and also for libraries that return async generators.

Merging several streams

You often have several sources and want items as soon as any produces one. A queue plus one task per source does it:

Python Merge async streams as items arrive

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output interleaves the sources in the order the items became ready (a0 a1 b0 a2 b1). A production version would use a bounded queue (backpressure) and take care of cancellation when the consumer stops early.

Pitfalls to avoid

  • yield inside async with or try blocks is allowed, but an abandoned generator keeps the resource open: use aclosing().
  • Awaiting inside __anext__ forever has no built-in timeout. Wrap the consumption in asyncio.timeout(), as in the timeouts lesson.
  • You cannot return value from an async generator (a bare return is fine).
  • Don't mix up await and async for. await gen() on an async generator is a TypeError; async generators are iterated, not awaited.
  • Sequential, not parallel. async for processes one item at a time. For concurrency use tasks, a TaskGroup or a queue with several workers.
  • Async generators are not thread-safe and should be consumed from one task at a time.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What makes a function an async generator?

    A function defined with async def that contains yield is an async generator; calling it gives an object you consume with async for.
  2. 2. What does the loop "async for x in stream" do between items?

    Each step awaits anext, so the loop can run other tasks while waiting for the next item.
  3. 3. Why wrap an async generator in contextlib.aclosing()?

    Leaving an async for early with break leaves the generator suspended. aclosing() closes it right away, so cleanup is deterministic.
  4. 4. Which statement about "async for" is true?

    async for is sequential. For concurrency use tasks or queues with several workers.

Summary

  • async for consumes async iterators; async generators (async def + yield) produce them; aiter()/anext() and async comprehensions are the companions.
  • Stack async generators to build memory-friendly pipelines such as filter, transform and batch.
  • Use aclosing() when you may stop early, and queues plus tasks to merge streams.
  • Iteration is sequential: reach for tasks when you need parallelism.

Next: Running blocking code from asyncio