Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 40 ยท Event Loop and Coroutines

Async Iterators, async for and Async Generators

Learn async for, __aiter__ and __anext__, async generators with yield, async comprehensions, aclosing and streaming data one item at a time in asyncio.

A normal for loop asks an iterator for the next item and gets it immediately. But sometimes the next item is not ready yet: the next line of a network stream, the next page of results, the next message in a queue. An ordinary loop would have to block. asyncio solves this with asynchronous iteration: async for can await between items, so the loop stays free to run other tasks while it waits.

async for and the async iterator protocol

An async iterable has an __aiter__ method returning an async iterator, which has an async def __anext__ method. __anext__ returns the next item or raises StopAsyncIteration when finished. You can write the class by hand:

Python An async iterator class

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

async for may only be used inside async def. Each pass awaits __anext__, so while a tick is pending, other tasks run.

Async generators: the easy way

Writing a class for every stream is tedious. An async generator is an async def function that contains yield. It produces an async iterator automatically:

Python An async generator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is the most common way to expose "a stream of values that take time to arrive": paginated API results, lines read from a socket, rows from a database cursor, changes from a file watcher.

Remember the difference: yield in a plain def makes a normal generator (use for); yield in an async def makes an async generator (use async for). An async generator cannot contain return value, only a bare return.

Pagination: a realistic example

Async generators hide paging details from the caller. Here a fake API returns pages of three items, and the generator keeps fetching until the pages run out:

Python Streaming paginated results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The caller sees one flat stream and can stop whenever it likes; the generator fetches pages only as needed. That is lazy streaming: memory stays small even if there are millions of records.

Async comprehensions

Comprehensions accept async for and await:

Python Async comprehensions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Careful: [await double(x) for x in ...] awaits one item at a time. To run them concurrently use asyncio.gather (gather lesson).

Cleaning up: aclose and aclosing

If you break out of an async for early, the generator is left suspended. Python eventually finalizes it, but the event loop does that later, which can delay cleanup such as closing a connection. For generators that hold resources, close them deterministically with contextlib.aclosing:

Python Deterministic cleanup with aclosing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

aclosing calls await generator.aclose() on exit, which raises GeneratorExit inside the generator at its yield so finally blocks run right away. Notice the cleanup message prints before "after the loop".

Handy built-ins and manual iteration

Python 3.10 added anext(iterator) and aiter(iterable):

Python
stream = aiter(countdown(3))
first = await anext(stream)         # 3
second = await anext(stream, None)  # a default avoids StopAsyncIteration

Python has no built-in async_map or async_filter; you write small async generators for those, or use libraries such as aiostream.

async for and concurrency

async for is sequential: it asks for the next item only after you finished the body of the loop. If you want to process items while the next ones are being fetched, feed a queue with a producer task. Async iteration is about not blocking the loop, not about running loop bodies in parallel.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What makes an async def function an async generator?

    yield inside async def creates an async generator, consumed with async for.
  2. 2. Which exception ends an async iterator?

    anext raises StopAsyncIteration to say there are no more items.
  3. 3. What does [x async for x in stream] need?

    Async comprehensions can only appear inside async functions (or an async-aware top level).
  4. 4. Why use contextlib.aclosing with an async generator?

    aclosing awaits aclose(), so cleanup code in the generator runs immediately instead of at some later garbage collection.

Summary

  • async for iterates objects with __aiter__ and async def __anext__, and may await between items.
  • Async generators (yield inside async def) are the easy way to build streams.
  • Async comprehensions use async for and await; awaits inside them are still sequential.
  • Use contextlib.aclosing when you may stop early and the generator holds resources.
  • async for does not make the loop body concurrent; use tasks or queues for that.

Next: Async context managers