Skip to content
Tutorials Center
My learning

Topics

Lesson 82 of 182 ยท Delegates, Events and LINQ

Iterators and yield

Learn how C# iterators work - yield return, yield break, the compiler state machine, lazy and infinite sequences, exceptions and finally in iterators, validation, and a preview of IAsyncEnumerable.

A normal method computes everything and hands back one result. An iterator method does something stranger: it hands back values one at a time, and between two values it simply stops and waits. The method is paused in the middle, with its local variables intact, until the caller asks for the next item. You write it with a single keyword, yield, and the compiler turns your method into a small machine that remembers where it was.

This idea is the engine behind every LINQ operator you have used, behind foreach over custom collections, and behind lazy pipelines that can process a file bigger than memory or a sequence that never ends. You met the basics in Indexers and Custom Iterators and Collection Interfaces. Here we go all the way down: what the compiler really generates, how laziness changes the order in which your code runs, what happens with exceptions, try/finally and early exits, and where IAsyncEnumerable<T> fits.

The shape of an iterator

A method is an iterator when it returns IEnumerable<T>, IEnumerator<T> (or their non-generic forms, or IAsyncEnumerable<T>) and contains at least one yield statement. There are only two forms of it:

  • yield return value; gives the caller the next item and pauses.
  • yield break; ends the sequence early (reaching the end of the method does the same).
C# Items arrive one at a time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Read the output as a conversation. The loop asks for the first item, the producer runs until the first yield return, hands over 1 and freezes. The consumer prints it. When the loop asks for the next item, the producer wakes up right after the yield return, prints "resumed", goes round the for loop and stops at the next yield return. The two sides take turns; nothing is ever stored in a list.

A method with yield cannot also return value;. Inside an iterator, return has no value; use yield break; to stop. The method signature promises a sequence, and the yield statements are how you supply it.

Nothing runs until you ask

The most important consequence of the pause-and-resume model is deferred execution: calling an iterator method does not execute a single line of its body. It only creates the object that will run the body later.

C# Calling the method runs nothing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

foreach is just sugar for this GetEnumerator / MoveNext / Current dance. Notice that "body starts" appears only after the first MoveNext(), and "body ends" only when the third MoveNext() runs off the end and returns false.

Laziness is a gift (you only pay for what you consume) and a trap (code you wrote runs later than you think, possibly several times, possibly never). The lesson on deferred execution explores the traps in LINQ queries. They all come from this one mechanism.

What the compiler really builds

You do not have to know the details to use yield, but knowing them removes all the mystery. For each iterator the compiler generates a hidden class that implements both IEnumerable<T> and IEnumerator<T>. Your local variables and parameters become fields of that class, and the method body becomes a MoveNext() method containing a switch on a hidden state number. Roughly:

TEXT
state -1  finished / disposed
state  0  not started yet
state  1  paused at the first yield return
state  2  paused at the second yield return
...

Each call to MoveNext() jumps to the right place, runs until the next yield return, stores the value in Current, records the new state and returns true. When the body ends it sets the state to finished and returns false. That is the whole trick, and it is why:

  • iterators are cheap: one small object per iteration, no thread, no coroutine runtime;
  • locals survive a pause (they are fields), so loops and if statements work naturally;
  • the sequence is a recipe: every foreach (every GetEnumerator() call) creates a fresh state machine that starts the body from the top with the same arguments.
C# Each foreach starts the body again

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The body ran twice because the sequence is a recipe, not stored data. If the body is expensive (reads a file, calls a service) and you need the items twice, materialize once with .ToList() or .ToArray().

Infinite sequences

Because items are produced on demand, a sequence is allowed to be infinite. The consumer decides when to stop, with break, Take, TakeWhile or First.

C# Fibonacci forever, taken lazily

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

while (true) looks like a hang, and it would be if you called ToList() on it. With Take(12) the consumer stops asking after twelve items, so the producer simply never resumes. The warning is real, though: any operator that needs the whole sequence (Count, ToList, OrderBy, Max, Reverse) will run forever or until memory ends. Document infinite sequences in the method name or comment.

yield break and early exit

yield break ends the sequence immediately, and is the iterator's version of return. It is useful for guard conditions and for stopping when you find a sentinel.

C# Stop at a sentinel with yield break

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

You can of course also just let the loop finish. yield break shines when you need to leave from the middle of nested loops or from a switch.

Writing your own LINQ operators

Since every LINQ operator is an iterator, writing one yourself is a good exercise and a handy tool. Here is Where, Select-like and a pairwise operator, built from the same ingredients as the real ones:

C# Three home-made operators

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

Notice Chunked yields bucket.ToArray(), a copy. Yielding bucket itself and then calling Clear() would let the consumer see its data wiped out if it kept the reference. Whenever an iterator reuses a buffer, think about who else can see it. (The BCL already has Chunk, and you can read how such operators are chained in LINQ Operators in Depth.)

Exceptions, validation and the lazy trap

Because the body is deferred, so are its exceptions. A validation check at the top of an iterator does not run when you call the method; it runs on the first MoveNext(), perhaps far from the bad call.

C# Validation fires late (fails on purpose)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The first line prints, and only then does the program crash with System.ArgumentOutOfRangeException: times ('-1') must be a non-negative value. raised from inside the loop. The fix is the wrapper pattern: an ordinary public method validates immediately, then returns a private iterator (often a local function).

C# Validate eagerly, iterate lazily

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The real LINQ methods work this way, which is why source.Where(null) throws at once even though the query is lazy.

Restrictions

The compiler has to be able to turn your method into a state machine, which limits what you may do:

  • No yield return inside a try block that has a catch clause (a try/finally is fine). The reason is that resuming into the middle of a protected region with a catch handler cannot be expressed by the generated state machine.
  • No yield in a finally block, in an anonymous function, or in an unsafe context.
  • No ref, out or in parameters on the iterator method. Since C# 13 you may use ref locals and ref struct values in an iterator, but only in stretches of code that contain no yield: they cannot stay alive across one.

If you need to catch errors from a producer, catch them in the consumer, or write the loop around MoveNext() by hand.

try/finally and cleaning up

A finally inside an iterator runs when the iteration ends, whether by finishing, by yield break, by an exception, or when the consumer stops early and disposes the enumerator. That last case is why foreach and using matter: foreach calls Dispose() on the enumerator for you, and Dispose() on an iterator runs the pending finally blocks.

C# finally runs even when the consumer quits early

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

This makes iterators a safe way to wrap resources: File.ReadLines works exactly like this. The corollary is a rule for consumers: if you drive an enumerator by hand, wrap it in using.

Delegating to another sequence

C# has no yield return* to splice another sequence in. You write a foreach and yield each element. That is simple but, for deep recursion, costs a hop per level per item, which is why recursive tree walks over huge trees are sometimes written with an explicit stack.

C# A recursive tree walk with yield

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

The result is a depth-first, pre-order walk, produced lazily: stop after the first match and the rest of the tree is never visited.

A preview: IAsyncEnumerable

When each item must be awaited (a page of results from a web service, a line from a network stream), you want the same pause-and-resume style but with await. An async iterator returns IAsyncEnumerable<T>, may use both await and yield return, and is consumed with await foreach (both C# 8). It is covered in depth in the async chapter.

C# An async iterator and await foreach

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with Console.ReadLine()), it gets the next line from this box.

When to use iterators, and when not

Use an iterator when you produce a sequence item by item and the consumer may not want all of it, when the sequence is large or infinite, when each item depends on the previous state, or when you want to wrap a resource safely. Return a List<T> or an array instead when the caller needs Count or indexing, will enumerate many times, or when the data already exists in memory (an iterator over a list just adds overhead). And never return a lazy sequence that reads shared mutable state the caller can change before enumerating: the surprise is yours to avoid.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What happens when you call a method that contains yield return, before any foreach?

    The call only creates the compiler-generated state machine. The body runs piece by piece as the consumer calls MoveNext, which is deferred execution.
  2. 2. Why does ArgumentOutOfRangeException.ThrowIfNegative at the top of an iterator fire late?

    The check is part of the deferred body. Wrap the iterator in a normal method (or use a local iterator function) so arguments are validated immediately.
  3. 3. Which of these is NOT allowed inside an iterator method?

    A try with a catch around a yield return cannot be turned into the state machine, so the compiler rejects it. try/finally is fine.
  4. 4. A consumer breaks out of a foreach over an iterator that has a try/finally. When does the finally block run?

    foreach calls Dispose on the enumerator, and disposing a paused iterator runs its pending finally blocks.
  5. 5. Which call on an infinite iterator would never finish?

    ToList needs every item, and an infinite sequence has none that is last. The other calls stop asking after a bounded number of items.

Summary

  • An iterator is a method with yield return / yield break that returns IEnumerable<T> or IEnumerator<T>; the compiler turns it into a state machine whose locals are fields.
  • Calling it runs nothing; each MoveNext() runs the body to the next yield return. This is deferred execution, and each new enumeration restarts the body.
  • Laziness allows infinite and memory-friendly sequences; operators that need everything (ToList, Count, OrderBy) do not mix with infinite ones.
  • Validate arguments in a non-iterator wrapper so errors surface at the call; exceptions inside the body appear at MoveNext().
  • No yield return in a try with catch, nor in finally or lambdas; try/finally is allowed and runs on disposal.
  • IAsyncEnumerable<T> with await foreach is the asynchronous twin.

Next: Nullable Reference Types