Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 48 Β· Enumerables and Data Processing

Lazy Processing with Stream

Learn Elixir Stream: lazy vs eager evaluation, Stream.map, take, iterate, unfold and cycle, streaming files line by line, and a JavaScript generator stand-in.

The Enum module is eager: every call walks the whole collection and builds a complete result before the next step begins. That is ideal for small data. It is a problem when the data is huge (a ten-gigabyte log file), expensive to produce, or infinite (all the natural numbers). For those jobs Elixir offers the Stream module, which processes elements lazily: one at a time, only when something asks for them.

A stream is a recipe, not a result. Building a stream does no work; the work happens when you finally run it with an Enum function. This lesson explains how that works, the most useful Stream functions, how to read files without loading them into memory, and how to think about streams using JavaScript generators.

Elixir cannot run on this site, so Elixir snippets are read-only. The runnable JavaScript examples use generators, which model the same lazy, one-at-a-time behavior. They are stand-ins, not Elixir.

Eager versus lazy

Look at this eager pipeline:

ELIXIR
1..1_000_000
|> Enum.map(fn n -> n * 3 end)
|> Enum.filter(fn n -> rem(n, 2) == 0 end)
|> Enum.take(3)
# [6, 12, 18]

Enum.map produces a list of a million elements, Enum.filter produces a list of half a million, and only then Enum.take keeps three. Change Enum to Stream for the first two steps:

ELIXIR
1..1_000_000
|> Stream.map(fn n -> n * 3 end)
|> Stream.filter(fn n -> rem(n, 2) == 0 end)
|> Enum.take(3)
# [6, 12, 18]

Same result, very different work. Each element travels through map and filter on its own, and as soon as Enum.take/2 has three results it stops asking for more. Only a handful of elements are ever touched.

Eager vs lazy

Eager (Enum) Each step finishes the whole collection first.
Lazy (Stream) Elements flow through all steps one by one.

Lazy steps do nothing until a final Enum call runs them.

1 / 2

Streams are recipes

A Stream function returns a value that describes the work but has not done it:

ELIXIR
stream = Stream.map([1, 2, 3], fn n ->
  IO.puts("processing #{n}")
  n * 2
end)
# nothing printed yet

Enum.to_list(stream)
# processing 1
# processing 2
# processing 3
# [2, 4, 6]

Inspecting stream in IEx shows something like #Stream<...>. Running it with Enum.to_list/1, Enum.take/2, Enum.reduce/3, Enum.sum/1 or any other Enum function is what triggers the computation. Running it twice repeats the work (and the side effects), because a stream is a description, not stored data.

Streams are lazy in a fused way: with several steps, the first element goes through all of them, then the second element, and so on. Compare the ordering of output:

JS Eager steps versus fused lazy steps

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Infinite streams: iterate, unfold, cycle, repeatedly

Because nothing is computed up front, a stream can be infinite as long as you take only a finite part:

ELIXIR
Stream.iterate(1, fn n -> n * 2 end) |> Enum.take(6)
# [1, 2, 4, 8, 16, 32]

Stream.cycle([:red, :green, :blue]) |> Enum.take(5)
# [:red, :green, :blue, :red, :green]

Stream.repeatedly(fn -> :rand.uniform(6) end) |> Enum.take(3)
# three random dice rolls

Stream.iterate/2 starts from a value and repeatedly applies a function. Stream.cycle/1 loops over a finite collection forever. Stream.repeatedly/1 calls a function again and again.

Stream.unfold/2 is the most flexible generator. You give it a starting state and a function that returns {element, next_state}, or nil to stop:

ELIXIR
fib =
  Stream.unfold({0, 1}, fn {a, b} -> {a, {b, a + b}} end)

Enum.take(fib, 10)
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

countdown = Stream.unfold(5, fn
  0 -> nil
  n -> {n, n - 1}
end)

Enum.to_list(countdown)
# [5, 4, 3, 2, 1]

Never call Enum.to_list/1, Enum.count/1 or Enum.sum/1 on an infinite stream: it will run until memory or time runs out. Always cut it down first with Enum.take/2, Stream.take/2, Stream.take_while/2 or Enum.find/2.

JS Infinite sequences with generators

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The Stream toolbox

Many Enum functions have lazy cousins in Stream:

Function What it does
Stream.map/2, filter/2, reject/2 transform or keep elements lazily
Stream.take/2, take_while/2, drop/2, drop_while/2 cut the stream lazily
Stream.with_index/1 pairs each element with its position
Stream.chunk_every/2 groups elements into lists
Stream.flat_map/2 map and flatten
Stream.uniq/1 drop duplicates as they pass
Stream.zip/2 pair two streams
Stream.scan/3 running accumulation, emitting every step
Stream.concat/2 join collections one after another
Stream.run/1 run a stream only for its side effects

For example, a running total with scan:

ELIXIR
[3, 1, 4, 1, 5]
|> Stream.scan(fn n, acc -> n + acc end)
|> Enum.to_list()
# [3, 4, 8, 9, 14]

Use Stream.take_while/2 to stop at the first element that fails a test, which is useful with an infinite source: Stream.iterate(1, &(&1 * 3)) |> Stream.take_while(&(&1 < 200)) |> Enum.to_list() gives [1, 3, 9, 27, 81].

Reading files lazily

The most common real-world use is files. File.read!/1 loads the whole file into memory. File.stream!/1 returns a stream of lines instead:

ELIXIR
"server.log"
|> File.stream!()
|> Stream.map(&String.trim/1)
|> Stream.filter(&String.contains?(&1, "ERROR"))
|> Stream.take(10)
|> Enum.to_list()

Only one line is held in memory at a time, and reading stops after ten error lines are found, even if the file has millions of lines. The stream opens the file when it starts running and closes it when it finishes or is cut short. File.stream! can also read fixed-size chunks of bytes instead of lines (check its docs for the argument order in your version), and a stream can be a destination too, because File.stream!/1 is collectable: Enum.into(lines, File.stream!("out.txt")) writes the lines to a file as they arrive.

For custom resources such as sockets or database cursors, Stream.resource/3 takes three functions: one to open the resource, one to produce the next elements, and one to clean up. It guarantees the cleanup runs even when the consumer stops early.

Pitfalls

  • Lazy does not mean faster. For small collections, Enum is usually quicker because streams add overhead per element. Use Stream when memory, infinity or early exit matter.
  • Nothing runs without a consumer. A pipeline that ends in Stream.map(...) and is never passed to Enum or Stream.run/1 does nothing, a common source of "why didn't my side effect happen" bugs.
  • Repeated runs repeat work. If you need a stream's results twice, run it once and keep the list, or you will recompute (and re-read the file).
  • Side effects happen in order but late. Do not rely on a log line appearing when you build the stream.

For the machinery behind both Enum and Stream, the Enumerable protocol, see Protocols and Behaviours.

Check your understanding

Quick check Β· 4 questions

of 4 answered You scored / 4 Β· perfect!

  1. 1. When does a pipeline built with Stream.map and Stream.filter actually do its work?

    Stream functions only describe the work. An Enum function such as to_list, take or reduce, or Stream.run/1, triggers it.
  2. 2. Which call is safe on an infinite stream?

    take/2 stops after five elements. The others try to consume the whole infinite stream and never finish.
  3. 3. What does the function passed to Stream.unfold/2 return to end the stream?

    unfold calls the function with the state and expects {element, next_state}, or nil to stop.
  4. 4. Why is File.stream!/1 better than File.read!/1 for a very large log file?

    File.stream!/1 produces lines lazily, so memory stays small and processing can end as soon as you have what you need.

Summary

  • Enum is eager; Stream is lazy and builds a recipe that runs only when consumed.
  • Lazy steps are fused: each element flows through all steps before the next one starts.
  • Stream.iterate/2, unfold/2, cycle/1 and repeatedly/1 create infinite streams; always limit them before collecting.
  • File.stream!/1 reads files line by line in constant memory and supports early exit.
  • Streams add per-element overhead, so use them for big, infinite or partially needed data, not by default.
  • A stream consumed twice does its work twice.

Next: Ranges, Sorting and Custom Ordering.