Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 149 ยท Modules and the Standard Library

itertools and functools

Master Python's itertools and functools. Chain, group, slice and combine iterables lazily, then use partial, reduce and cache to build smarter functions.

Two small modules hold a surprising amount of power. itertools is a toolbox of building blocks for iterators: chaining sequences, grouping, slicing endless streams, and generating every combination of some items. functools works on functions themselves: freezing some arguments, folding a list into one value, and remembering results so slow functions only run once per input.

Both modules let you replace hand-written loops with short, well-tested calls. Most itertools functions are lazy: they produce items one at a time on demand, like generators, so they work even on huge or infinite inputs.

Joining and slicing iterables

chain() walks through several iterables one after another, without building a new list. islice() takes a slice of any iterator, even one that never ends:

Python chain and islice

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

count() and cycle() are infinite, so never call list() on them directly. Pair them with something that stops: islice(), zip() with a finite iterable, or a loop with break.

Neighbors and batches

Two newer helpers handle very common loops. pairwise() (Python 3.10+) gives each item with its successor; batched() (Python 3.12+) splits data into fixed-size chunks:

Python pairwise and batched

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last batch can be shorter than the rest. In Python 3.13+, batched(guests, 3, strict=True) raises an error instead, for cases where an incomplete chunk means bad data.

Running totals with accumulate

accumulate() yields running results: by default a running sum, but you can pass any two-argument function:

Python accumulate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The operator module provides ready-made functions for Python's operators (operator.add, operator.mul, operator.itemgetter and more), so you don't need tiny lambdas.

Grouping with groupby

groupby() collects consecutive items that share a key. It's like the "group by" in spreadsheets, with one catch you must remember:

Python groupby (sort first!)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

groupby() only groups neighbors. If equal keys aren't next to each other, you get several separate groups. Sort by the same key first, or use a defaultdict(list) from collections when the data isn't sorted. Also, each group is a one-time iterator: consume it before moving to the next group.

groupby() is also great for spotting runs: [(k, len(list(g))) for k, g in groupby("aaabccdddd")] gives [('a', 3), ('b', 1), ('c', 2), ('d', 4)], a simple run-length encoding.

Combinatorics: product, permutations, combinations

These functions generate every arrangement of some items, which replaces deeply nested loops:

Python Every combination

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Function Order matters? Repeats an item? Items from "ABC", length 2
product(..., repeat=2) yes yes 9
permutations(..., 2) yes no 6
combinations(..., 2) no no 3
combinations_with_replacement(..., 2) no yes 6

The counts grow very fast. permutations(range(12)) has nearly half a billion results, so keep these lazy and stop early when you find what you need.

Other useful itertools functions include takewhile() and dropwhile() (stop or start at a condition), zip_longest() (zip uneven iterables with a fill value), starmap() and tee().

Laziness: a double-edged sword

Because itertools functions are lazy, they return iterators, not lists. That has three practical consequences:

  • Printing one shows something like <itertools.chain object at 0x...>. Wrap it in list() when you want to see the items.
  • An iterator can be consumed only once. Loop over a chain() result twice and the second loop finds nothing. Store the items in a list if you need them again.
  • Laziness saves memory. sum(islice(huge_iterator, 1000)) only ever holds one item at a time, no matter how big the source is.

functools.partial: pre-filling arguments

partial() creates a new function with some arguments already filled in. It's a clean way to adapt a general function to a specific job:

Python partial

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Partials are especially handy as callbacks and as the key or function argument for higher-order functions like map() and sorted().

functools.reduce: folding a sequence

reduce() combines items pairwise from left to right until one value remains: reduce(f, [a, b, c]) computes f(f(a, b), c).

Python reduce

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Before reaching for reduce(), check for a built-in: sum(), max(), min(), any(), all(), math.prod() and "".join() cover most cases and read better. max(words, key=len) beats the lambda above.

functools.cache: remember results

@cache is a decorator that stores each result by its arguments. Call the function again with the same arguments and you get the saved answer instantly. That turns slow recursive functions into fast ones, as you saw in Recursion:

Python cache and lru_cache

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • @cache (Python 3.9+) keeps every result forever. Simple and fast.
  • @lru_cache(maxsize=n) keeps only the n most recently used results, so memory stays bounded.
  • Arguments must be hashable (numbers, strings, tuples), because they become dictionary keys. A list argument raises TypeError.
  • Only cache pure functions, whose result depends only on their arguments. Caching something that reads the clock or a file returns stale answers.

Two more functools tools you'll meet: @wraps keeps a wrapped function's name and docstring when writing decorators, and @cached_property computes an attribute once per object on first access.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does list(chain([1, 2], (3,), "ab")) return?

    chain() yields every item of each iterable in turn. A string is iterable, so it contributes its characters.
  2. 2. groupby() on the list ["a", "b", "a"] gives three groups. Why?

    Sort by the same key first so equal items sit next to each other.
  3. 3. How many results does combinations("ABCD", 2) produce?

    Combinations ignore order and never repeat an item: AB, AC, AD, BC, BD, CD. permutations would give 12.
  4. 4. What does partial(round, ndigits=1)(3.14159) return?

    The partial pre-fills ndigits=1, so calling it with 3.14159 runs round(3.14159, ndigits=1).
  5. 5. Which function is a poor candidate for @cache?

    The time changes, so a cached result would be stale. Cache only pure functions whose result depends only on their arguments.

Summary

  • itertools functions are lazy building blocks: chain() joins, islice() slices, count() and cycle() never end.
  • pairwise() pairs neighbors, batched() makes chunks, accumulate() produces running totals.
  • groupby() groups consecutive items, so sort by the same key first.
  • product(), permutations() and combinations() replace nested loops for arrangements.
  • partial() pre-fills arguments, reduce() folds a sequence (prefer built-ins when they exist), and @cache / @lru_cache remember results of pure functions.

Next: Working with JSON, where you'll turn Python data into JSON text and back.