Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 134 ยท Methods and Blocks

Enumerators and Lazy Enumerators

Learn Ruby Enumerator objects - external iteration with next and peek, Enumerator.new, to_enum, Enumerator.produce, and Enumerator::Lazy for infinite sequences.

When you call each, map or select without a block, Ruby does not complain. It hands you back an Enumerator: an object that represents the iteration without running it yet. That small idea unlocks a lot: you can chain each_with_index.map, step through a collection one element at a time with next, describe an infinite sequence such as "all Fibonacci numbers", and process only the part you need with lazy enumerators.

This lesson explains the difference between internal iteration (the collection drives, as with each { }) and external iteration (you drive, with next), shows how to create enumerators from scratch with Enumerator.new, to_enum and Enumerator.produce, and introduces lazy for efficient pipelines on large or infinite data.

Enumerators come from blocks-less iterators

Ruby What you get without a block

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Array#each without a block returns #<Enumerator: [10, 20, 30]:each>, which names the receiver and the method. The class Enumerator includes Enumerable, so it has map, select, to_a and the rest. with_index and each_with_index are enumerator methods that add a counter to any iteration, which is why map.with_index gives you a mapping with an index, something map alone cannot do.

Internal versus external iteration

With each { } the collection is in charge: it calls your block for every element, and you cannot pause in the middle. An enumerator can also be driven by you: next returns the next element, and peek shows it without consuming it.

Ruby Stepping with next and peek

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

After the last element, next raises StopIteration with the message iteration reached an end. Ruby's loop method rescues StopIteration automatically and ends quietly, so you can write a clean external loop:

Ruby loop catches StopIteration

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

External iteration is the only way to walk two collections in lockstep or to pull items one at a time from something that produces them lazily, such as a parser reading tokens. Internally, an enumerator that you step with next runs the iteration in a separate fiber so that it can pause between elements.

Building your own: Enumerator.new

Enumerator.new takes a block that receives a yielder. Every yielder << value (or yielder.yield value) produces one element, and the block is paused until the consumer asks for the next one. The generator can therefore be infinite.

Ruby An infinite Fibonacci sequence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The block never finishes, yet take(10) returns promptly: take asks for ten values and stops. Note the lazy on the second to last line. Without it, fib.select(&:even?) would try to build the whole (infinite) result and never return. We come back to this below.

Never call to_a, sort, map, select or size-dependent methods on an infinite enumerator without lazy, first(n) or take(n). The program will run until it is killed or runs out of memory. In the runner here, it would hit the 5 second limit.

Enumerators as method return values

A method that yields can return an enumerator when no block is given, with to_enum (or its alias enum_for). This is the idiom that makes your own classes feel like built-ins.

Ruby to_enum in your own class

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Passing a block to to_enum ({ @songs.size }) tells the enumerator how to compute its size without iterating. Using __method__ avoids repeating the method name. Whenever you write an iterator, add the return to_enum(...) unless block_given? line; callers will thank you.

Enumerator.produce

Enumerator.produce(initial) { |previous| next_value } builds an infinite sequence from a seed and a rule for getting the next value. The sequence ends if the block raises StopIteration.

Ruby Sequences from a seed and a rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

produce is the functional replacement for a while loop that keeps updating a variable. The Collatz example is a good illustration: starting from 27 the sequence has 112 values (111 steps, so count includes both the 27 and the final 1) and climbs to 9232 before falling to 1, and we described it in six lines without a counter or an accumulator. The chain-walking example, "follow parent until there is none", is common in tree and graph code.

Lazy enumerators

A normal enumerator chain is eager: each step finishes for all elements before the next step starts. Look at the order of events:

Ruby Eager versus lazy order of work

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The eager version performs 12 steps (six maps, then six selects) to produce one answer. The lazy version pushes each element through the whole pipeline before touching the next one, and stops as soon as first(1) has its answer: only six steps, and the log shows that the pipeline gives up after element 3 produced 6. For small arrays nobody notices. For large or infinite inputs the difference is between working and not working.

Call lazy to start a lazy chain; the methods map, select, reject, take, take_while, drop, drop_while, flat_map, zip, uniq, filter_map, with_index and compact stay lazy. To get values out, call first(n), to_a, force, each or any other method that needs results.

Ruby Lazy pipelines on infinite input

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

(2..) is an endless range, so there is no end to the chain. Printing long.class shows Enumerator::Lazy; nothing was computed until first(2). Use eager when you want to switch back to normal behavior.

When lazy helps and when it hurts

  • Use it for infinite sources, very large files read line by line (File.foreach(...).lazy), and pipelines where you stop early with first, take or find.
  • Avoid it for small collections or when you need every element anyway. Each lazy step wraps the element in extra machinery, so a lazy pipeline over a short array is slower than the eager one.
  • Lazy chains read well but can hide side effects in the order the steps run. Keep blocks pure.

Enumerator::Yielder, size and chaining details

Some facts worth knowing:

Ruby Small details that matter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The first line shows why size is often nil: Ruby cannot know how many values the generator will produce without running it. Starting an enumerator with next creates a fiber, and fibers cannot be duplicated, hence the message from the rescue modifier (see Fibers).

Pitfalls

  • next and side effects: an enumerator you step with next runs ahead in its own fiber. Do not rely on side effects of the generating block happening at a particular moment.
  • StopIteration outside loop: you must rescue it yourself or use loop.
  • Infinite sequences: always bound them with first, take, take_while or lazy.
  • Reusing an enumerator: after next consumed some elements, methods such as map still start from the beginning; next has its own cursor. Call rewind to reset it.
  • Mutating the collection while an enumerator walks it gives surprising results; iterate over a copy.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does [1, 2, 3].each (with no block) return?

    Built-in iterators return an Enumerator when no block is given, which lets you chain with_index, next, lazy and more.
  2. 2. What happens when you call next on an enumerator that has no more elements?

    StopIteration is raised (message: iteration reached an end). The loop method rescues it for you.
  3. 3. Which expression returns the first five squares of the natural numbers without an endless loop?

    map on an endless range is eager and never finishes. lazy makes the chain process one element at a time until first(5) has what it needs.
  4. 4. What does Enumerator.produce(1) { |n| n * 2 }.take(4) return?

    produce starts with the seed 1 and applies the block to the previous value each time: 1, 2, 4, 8.
  5. 5. Why should your own iterator methods start with return to_enum(:name) unless block_given??

    Returning an enumerator when no block is given matches the built-in iterators and enables with_index, next, lazy and other tools.

Summary

  • An Enumerator represents an iteration; built-in iterators return one when called without a block.
  • Use next, peek and rewind for external iteration; loop handles StopIteration.
  • Create enumerators with Enumerator.new { |y| y << x }, to_enum in your own iterators and Enumerator.produce(seed) { }.
  • Infinite enumerators need first, take, take_while or lazy.
  • lazy pushes each element through the whole pipeline and stops early; it pays off on big or infinite data, not on small arrays.
  • Stepping with next uses a fiber behind the scenes, which is the topic of the next lesson.

Next: Fibers