Skip to content
Tutorials Center
My learning

Topics

Lesson 103 of 201 ยท Generics and Functional Programming

Advanced Streams

Go beyond stream basics - flatMap and mapMulti, takeWhile/dropWhile, infinite streams, parallel streams and when to avoid them, Java 24 gatherers and common mistakes.

Once filter, map and collect feel natural, you'll start to run into problems they can't express cleanly. What if each element should turn into several elements, like orders into their order lines? How do you stop a stream at the first element that breaks a rule, or process a sequence that never ends? When should you make a stream parallel, and when does that make things worse? And how do you write a step the standard library doesn't have, such as "group into batches of three" or "running total"?

This lesson answers those questions: flattening with flatMap and mapMulti, the ordered slicing operations takeWhile and dropWhile, infinite streams with iterate and generate, a few source subtleties, parallel streams, the gatherers added in Java 24, and the mistakes that experienced developers still make. It builds on Streams and Collecting Stream Results.

flatMap: one element in, many out

map produces exactly one output per input. flatMap takes a function that returns a stream for each element and concatenates all those streams into one. Think "map, then flatten":

TEXT
map:      [order1, order2]  ->  [[lineA, lineB], [lineC]]   (a stream of lists)
flatMap:  [order1, order2]  ->  [lineA, lineB, lineC]        (one flat stream)
Java Flattening nested data with flatMap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

An empty inner stream simply contributes nothing, which is why the empty list in grid disappears. There are primitive versions too: flatMapToInt, flatMapToLong and flatMapToDouble.

mapMulti: flatMap without the inner streams

mapMulti (Java 16) does the same job in a different style. Instead of returning a stream, your function receives each element plus a consumer, and you call consumer.accept(...) zero, one or many times. It avoids creating a small stream per element, and it reads well when the logic has ifs or loops:

Java mapMulti: emit zero, one or many results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The explicit type witness .<Integer>mapMulti(...) tells the compiler the output type, which it often can't infer from a lambda alone. Rule of thumb: use flatMap when you already have a collection or stream to return; use mapMulti when you'd otherwise build a throwaway list just to stream it.

takeWhile and dropWhile

Java 9 added two slicing operations driven by a condition instead of a count:

  • takeWhile(predicate) passes elements while the predicate is true, and stops at the first false.
  • dropWhile(predicate) skips elements while the predicate is true, then passes everything after, without checking again.

Unlike filter, they care about position. filter(n -> n < 5) keeps every small number anywhere; takeWhile(n -> n < 5) stops at the first big one. That makes them ideal for sorted data or sequences like log lines, where "the rest" means something.

Java takeWhile vs filter, and dropWhile

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Infinite streams: iterate and generate

A stream doesn't need a finite source. Two factory methods create potentially infinite ones:

  • Stream.iterate(seed, next) produces seed, next(seed), next(next(seed)), ...
  • Stream.iterate(seed, hasNext, next) (Java 9) works like a for loop and stops when hasNext is false.
  • Stream.generate(supplier) calls the supplier for each element.

Because pipelines are lazy and pull elements one by one, infinite streams are fine as long as something short-circuits: limit, takeWhile, findFirst or anyMatch. Without that, the terminal operation never finishes.

Java iterate and generate with a stopping point

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Order matters with infinite streams. Stream.iterate(1, n -> n + 1).filter(n -> n < 5).limit(10) never ends: only four elements ever pass the filter, so limit(10) waits forever. sorted() and distinct() on an infinite stream can also hang or run out of memory, since they need to see everything first.

Source subtleties: Stream.of vs Arrays.stream

Stream.of(T... values) is a varargs method, and that leads to a classic surprise with primitive arrays. An int[] is a single object, not a list of Ts, so Stream.of(intArray) creates a stream with one element: the array itself.

Java
int[] numbers = { 3, 1, 2 };
Stream.of(numbers).count();       // 1, a Stream<int[]>
Arrays.stream(numbers).count();   // 3, an IntStream
IntStream.of(numbers).count();    // 3, an IntStream

String[] names = { "a", "b" };
Stream.of(names).count();         // 2, object arrays spread into varargs

Prefer Arrays.stream(array) for arrays; it also has an overload for a sub-range: Arrays.stream(array, from, to). A few more useful sources: Stream.ofNullable(x) (Java 9) gives an empty stream for null and a one-element stream otherwise, Stream.concat(a, b) joins two streams, and Pattern.compile(",").splitAsStream(text) streams the parts of a string (see Regular Expressions).

Parallel streams, and when not to use them

Call parallelStream() on a collection, or .parallel() on any stream, and the work is split across the CPU cores using the common Fork/Join pool. The result is the same as the sequential version if your pipeline follows the rules: no shared mutable state, associative reductions, and no reliance on side-effect order.

Java Parallel streams: same result, different order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Results are collected in encounter order, but the processing happens in whatever order the threads pick. Plain forEach on a parallel stream prints in an unpredictable order; forEachOrdered restores it at a cost.

When parallel streams hurt more than they help:

  • Small inputs. Splitting and merging cost more than the work itself. Thousands of cheap operations are usually faster sequentially.
  • Shared mutable state. parallelStream().forEach(list::add) on an ArrayList loses elements or throws, because ArrayList isn't thread-safe. Use collect or toList.
  • Blocking work. Parallel streams share one common pool across the whole JVM. Doing I/O or calling slow services inside them starves every other parallel stream. Use an executor or virtual threads instead.
  • Order-sensitive or stateful steps. limit, findFirst, sorted and distinct must coordinate between threads to respect order, which can erase the gain. If order doesn't matter, findAny and .unordered() help.
  • Sources that split badly. ArrayList, arrays and IntStream.range split cheaply. LinkedList, Stream.iterate and I/O-backed streams don't.

The honest rule: start sequential, and only try parallel() for CPU-heavy work on large data, after measuring both versions.

Gatherers: custom intermediate operations (Java 24)

Collectors let you write custom terminal logic, but for years there was no way to add a custom intermediate operation. Java 24 finalized stream gatherers: stream.gather(gatherer) runs any Gatherer as a pipeline step. The Gatherers class ships five ready-made ones:

Gatherer What it does
Gatherers.windowFixed(n) Groups elements into lists of n (the last may be shorter)
Gatherers.windowSliding(n) Overlapping windows: [1,2,3], [2,3,4], ...
Gatherers.scan(init, f) Running results: emits every intermediate value of a fold
Gatherers.fold(init, f) One final value, emitted as a single-element stream
Gatherers.mapConcurrent(n, f) Maps with up to n concurrent virtual threads, keeping order
Java Built-in gatherers: windows, scan and fold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

You can also write your own gatherer. Gatherer.ofSequential(initializer, integrator) takes a supplier for per-stream state and an integrator that receives the state, each element and a downstream to push results to. Returning false from the integrator stops the stream early, just like limit.

Java A custom gatherer: distinct by key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

distinct() can only compare whole elements with equals; this gatherer deduplicates by any property while keeping the first occurrence. Gatherers are final in Java 24 and later; on Java 22โ€“23 they were a preview feature, and on older versions you'd write a loop or a collector.

Common mistakes and how to avoid them

  • Using map when you need flatMap. If you end up with Stream<List<X>> or Stream<Stream<X>>, you wanted flatMap.
  • Calling get() on an Optional from findFirst. Use orElse, orElseThrow or ifPresent instead (Optional).
  • Checked exceptions in lambdas. A lambda passed to map can't throw a checked exception like IOException. Wrap it in an unchecked exception (UncheckedIOException) or move the risky code into a method that handles it.
  • Huge, clever pipelines. A stream of 15 operations with nested collectors is hard to debug. Extract steps into well-named methods or intermediate variables, and use a loop when it's clearer.
  • Assuming streams are faster. They are about clarity. Boxing in Stream<Integer> can be noticeably slower than IntStream or a loop in hot code. Measure before you optimize.
  • Not closing I/O streams. Streams from Files.lines(path) hold an open file. Use them in try-with-resources: try (var lines = Files.lines(path)) { ... } (Reading and Writing Files).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does Stream.of(List.of(1, 2), List.of(3)).flatMap(List::stream).count() return?

    flatMap concatenates the inner streams into one stream of three elements, 1, 2 and 3.
  2. 2. What does Stream.of(1, 2, 7, 3).takeWhile(n -> n < 5).toList() return?

    takeWhile stops at the first element that fails the predicate (7) and never looks at the 3. filter would return [1, 2, 3].
  3. 3. With int[] a = {1, 2, 3};, what does Stream.of(a).count() return?

    A primitive array is one object, so Stream.of(a) is a Stream of int arrays with a single element. Use Arrays.stream(a) to get an IntStream of 3 values.
  4. 4. Which is the best candidate for a parallel stream?

    Parallel streams help with large, CPU-bound, independent work on sources that split well. Blocking I/O, shared mutable lists and order-sensitive output are poor fits.
  5. 5. What does Stream.of(1, 2, 3, 4, 5).gather(Gatherers.windowFixed(2)).toList() return?

    windowFixed groups elements into non-overlapping lists of the given size, and the last window holds whatever is left. windowSliding would produce the overlapping version.

Summary

  • flatMap turns each element into a stream and flattens the results; mapMulti does the same by pushing values to a consumer, without inner streams.
  • takeWhile stops at the first non-matching element; dropWhile skips a leading run and keeps everything after.
  • Stream.iterate and Stream.generate create infinite streams; always bound them with limit, takeWhile or a short-circuiting terminal operation.
  • Use Arrays.stream for primitive arrays: Stream.of(intArray) has one element.
  • Parallel streams pay off only for large, CPU-bound, side-effect-free work on well-splitting sources; measure first and never do blocking I/O in them.
  • Java 24 gatherers (stream.gather(...)) add custom intermediate steps; Gatherers provides windowFixed, windowSliding, scan, fold and mapConcurrent.

Next: Optional, the type that stream operations like findFirst and max return, and how to use it to avoid null.