Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 149 ยท Functions

Generators

Write Python generator functions with yield, build lazy and infinite sequences, use generator expressions to save memory, chain pipelines and delegate with yield from.

A normal function computes its result, returns it and forgets everything. A generator function is different: it produces a series of values one at a time, pausing after each one and resuming exactly where it left off when the next value is requested. You write one by using yield instead of (or along with) return.

Generators are how Python handles sequences that are huge, endless, or expensive to compute. Instead of building a list of a million items in memory, a generator hands them over as they're needed. In this lesson you'll learn how yield works, how to build lazy pipelines, how generator expressions compare to list comprehensions, and how yield from delegates to another generator.

Your first generator

Any function that contains yield becomes a generator function. Calling it doesn't run the body. Instead, it returns a generator object, and the body runs a piece at a time as you ask for values:

Python yield pauses the function

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Trace the output carefully:

  1. countdown(3) creates the generator but prints nothing.
  2. The first next(gen) runs the body until the first yield, which hands 3 back and freezes the function, local variables and all.
  3. The next next(gen) resumes right after that yield, loops once and yields 2.
  4. The for loop keeps resuming it until the body ends, at which point the generator signals that it's exhausted.

Generators are single-use

A generator remembers its position, and once it has produced everything, it's done. Looping over it again gives nothing. Calling next() on an exhausted generator raises StopIteration, which is the signal for loops use to know when to stop.

Python Exhaustion

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If you need to go through the values more than once, either call the generator function again or store the values in a list.

Why lazy evaluation matters

A generator computes each value only when asked. That gives you three superpowers:

  • Memory: values exist one at a time, so you can process millions of items with a tiny, constant amount of memory.
  • Speed to first result: work starts immediately; you don't wait for an entire list to be built.
  • Infinite sequences: a generator can yield forever, because it only produces what you consume.
Python Infinite and lazy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

itertools.islice takes a slice of any iterator, including an infinite one. The size comparison shows the list holding a million references, while the generator object stays tiny no matter how many values it will produce. (getsizeof measures only the container itself, but the difference is the point.)

Generator expressions

You've seen the generator in the last example: (n * n for n in range(1_000_000)). A generator expression looks like a list comprehension in parentheses, and it creates a generator without a def.

Python
squares_list = [n * n for n in range(10)]   # builds a list now
squares_gen = (n * n for n in range(10))    # produces values on demand

When a generator expression is the only argument to a function, you can drop the extra parentheses:

Python
total = sum(price * qty for price, qty in cart)
longest = max(len(line) for line in lines)
has_admin = any(user.is_admin for user in users)

Use a generator expression when you'll consume the values once, especially when feeding sum, max, min, any, all, "".join() or a for loop. Use a list comprehension when you need to index the result, loop over it twice or check its length. The Dictionary and Set Comprehensions lesson covers the related forms.

Generator pipelines

Small generators chain together like stages on an assembly line. Each stage pulls one item at a time from the previous one, so the whole pipeline processes a stream without ever holding all of it in memory. This is a natural way to handle log files and large data sets.

Python A processing pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

No stage knows about the others; each just consumes an iterable and yields results. You could swap the file for a network stream or add a new stage without changing the rest. (Reading and Writing Files and Paths with pathlib cover the file parts.)

Delegating with yield from

When a generator wants to yield every value from another iterable, yield from does it in one line. It's both shorter than a for loop and handy for recursive generators:

Python yield from

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare flatten with the recursive total_size in Recursion: the generator version doesn't need to build and merge lists at every level; it streams values straight to the caller.

return inside a generator

A generator can use a plain return to stop early. Any value given to return doesn't come out of the loop; it's attached to the StopIteration exception, and yield from receives it as its result. That's an advanced feature you'll rarely need, so for now remember: inside a generator, return simply means "I'm done".

Python
def take_until_blank(lines):
    for line in lines:
        if not line:
            return          # stop the generator
        yield line

Note: Generators can also receive values with gen.send(value) and be closed early with gen.close(). These features power coroutine-style code and contextlib.contextmanager (see Context Managers), but everyday generator code only needs yield.

When a list is the better choice

Generators aren't always the answer. Pick a list when you need to:

  • loop over the values more than once, or in more than one place;
  • index, slice, sort or reverse them, or ask for len();
  • show or debug the values (printing a generator only shows <generator object ...>);
  • keep a small result around, where memory doesn't matter.

A useful habit is to write the producing code as a generator and let the caller decide: list(read_records()) when they need everything at once, or a plain for loop when they want to stream.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What happens when you call a generator function, like gen = countdown(3)?

    The body starts running only when you ask for the first value, with next() or a for loop.
  2. 2. After g = (x for x in range(3)) and list(g), what does a second list(g) return?

    Generators are single-use. Once exhausted, they produce nothing more.
  3. 3. Which is the better choice when you only need the total of a million computed values?

    The generator expression feeds sum one value at a time instead of building a million-item list first.
  4. 4. What does yield from other do inside a generator?

    It delegates to the other iterable, equivalent to looping over it and yielding each item (plus passing through send/return values).
  5. 5. How can you safely take the first 5 values from an infinite generator?

    list(gen) would never finish, and generators don't support indexing or len(). itertools.islice stops after 5 values.

Summary

  • A function containing yield is a generator function; calling it returns a generator object without running the body.
  • Each next() resumes the body until the next yield; local state is kept between values. When the body ends, StopIteration tells loops to stop.
  • Generators are lazy and single-use: they save memory, start producing results immediately and can represent infinite sequences.
  • Generator expressions (expr for x in items) are the lazy cousins of list comprehensions; pass them straight to sum, max, any and friends.
  • Chain small generators into pipelines to process streams one item at a time.
  • yield from iterable delegates to another iterable, which is ideal for composing and recursive generators.

Next: Iterators and the Iteration Protocol, the mechanism behind for loops, generators and every iterable in Python.