Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 37 ยท DataStream API

keyBy, Reduce and Rolling Aggregations

Group streams by key and aggregate them in Flink: keyBy selectors, rolling sum, min, max, minBy and maxBy, reduce, and how to pick keys safely, with runnable examples.

Counting per user, summing per product, finding the highest temperature per sensor: all of these start with the same step, keyBy, and then use an operator that remembers one running value for every key. These are called rolling aggregations because they emit an updated result for every incoming record instead of waiting for the end.

This lesson covers keyBy, the built-in aggregations and reduce, and the details that surprise beginners. It follows Map, Filter and FlatMap.

keyBy

Java
DataStream<Tuple2<String, Integer>> pairs = ...;
KeyedStream<Tuple2<String, Integer>, String> keyed = pairs.keyBy(t -> t.f0);

DataStream<Order> orders = ...;
KeyedStream<Order, String> byCustomer = orders.keyBy(Order::customerId);

keyBy takes a key selector function that extracts the key. It does not change records; it routes them so that all records with the same key reach the same subtask (see Parallelism, keyBy and Key Groups). The result is a KeyedStream, which unlocks keyed state, rolling aggregations, windows and KeyedProcessFunction.

Key rules again: the key type needs a stable hashCode() and equals(), so avoid arrays and enums. A composite key can be a Tuple2 or a record:

Java
record RegionProduct(String region, String productId) {}
orders.keyBy(o -> new RegionProduct(o.region(), o.productId()));

Built-in rolling aggregations

On a KeyedStream of tuples you can use field positions, and on POJOs field names:

Java
keyed.sum(1);          // running sum of field 1 per key
keyed.min(1);          // running minimum of field 1 (other fields: from the FIRST record of the key)
keyed.max(1);
keyed.minBy(1);        // the whole record that has the minimum so far
keyed.maxBy(1);

The distinction between min and minBy matters. min(1) returns the minimum value of field 1, but the other fields of the emitted record are not defined (they usually come from the first record of the key). minBy(1) returns the whole record that holds the smallest value, so related fields belong together. When in doubt, use minBy/maxBy.

For every incoming record, the stream emits the new aggregate of its key, so the output has as many records as the input.

reduce

reduce is the general rolling aggregation. Your function combines the previous result of the key with the new record and returns a value of the same type:

Java
DataStream<Order> totals = orders
    .keyBy(Order::customerId)
    .reduce((acc, next) -> new Order(acc.customerId(), acc.amount() + next.amount()));

Properties of a good reduce function: it must be associative and should not depend on the arrival order, because after a restart records are replayed and with parallel sources the order across subtasks is not guaranteed. State per key is a single value of the record type, so memory is bounded by the number of keys. If your accumulator has a different type from the input (such as count and sum for an average), use AggregateFunction with windows or a KeyedProcessFunction with ValueState; see Process Functions.

Stand-in: rolling aggregations by hand

Java Stand-in: sum, min, minBy and reduce per key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Each input produces one output line with the updated aggregate for its key. Note how minBy keeps the time that belongs to the minimum temperature, which a plain min would not guarantee.

Averages need a different accumulator

An average cannot be a plain reduce of doubles, because averaging averages is wrong. Keep (sum, count) and divide at the end:

Java
public class Avg implements AggregateFunction<Reading, Tuple2<Double, Long>, Double> {
    public Tuple2<Double, Long> createAccumulator() { return Tuple2.of(0.0, 0L); }
    public Tuple2<Double, Long> add(Reading r, Tuple2<Double, Long> acc) { return Tuple2.of(acc.f0 + r.temp(), acc.f1 + 1); }
    public Double getResult(Tuple2<Double, Long> acc) { return acc.f0 / acc.f1; }
    public Tuple2<Double, Long> merge(Tuple2<Double, Long> a, Tuple2<Double, Long> b) { return Tuple2.of(a.f0 + b.f0, a.f1 + b.f1); }
}

AggregateFunction is used with windows (.window(...).aggregate(new Avg())) and is the building block of incremental aggregation.

Java Stand-in: why averaging averages is wrong

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Rolling aggregates grow forever

A rolling aggregation on an unbounded stream keeps one value for every key it ever saw. Millions of one-time user IDs mean millions of state entries that are never removed, and the job eventually runs out of memory or disk. Mitigations:

  • Use windows so state is dropped when a window closes (Tumbling and Sliding Windows).
  • Configure state TTL to expire idle keys.
  • Use a KeyedProcessFunction with a timer that clears the state after inactivity.
  • Choose a key with a bounded number of values.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a rolling sum on a KeyedStream emit?

    Rolling aggregations emit one updated aggregate per input record.
  2. 2. How does minBy differ from min?

    With min only the compared field is guaranteed; minBy keeps the matching record intact.
  3. 3. Which key type is a bad choice for keyBy?

    Arrays have identity-based hash codes, so equal contents do not give the same key.
  4. 4. Why is an average not a plain reduce over doubles?

    Use AggregateFunction or a process function with an accumulator of a different type.

Summary

  • keyBy routes records by a stable key and gives a KeyedStream.
  • sum, min, max, minBy, maxBy and reduce keep one value per key and emit an update for each record; prefer minBy/maxBy when other fields matter.
  • Use accumulators with a different type (AggregateFunction) for averages.
  • Rolling state grows with the number of keys: bound it with windows, TTL or timers.

Next: Process Functions.