Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 37 ยท DataStream API

Map, Filter and FlatMap

Learn Flink's basic stateless transformations: map, filter, flatMap, lambdas versus function classes, rich functions with open and close, and how to handle bad records.

Most of a real pipeline is made of three small stateless operations: change each record (map), drop some records (filter), or turn one record into zero, one or many (flatMap). They are the "SELECT and WHERE" of streaming, cheap to run, and they can be chained into one task as explained in The Dataflow Model.

This lesson shows each one with real Flink code and a runnable stand-in, then covers rich functions for set-up work and techniques to deal with broken input.

map: one in, one out

map applies a function to every record and emits exactly one result.

Java
DataStream<String> names = env.fromData("ana", "bo", "cy");
DataStream<Integer> lengths = names.map(String::length);
DataStream<String> shout = names.map(n -> n.toUpperCase() + "!");

Alternatives to a lambda: an anonymous class or a named class implementing MapFunction<IN, OUT>:

Java
public class ToFahrenheit implements MapFunction<Double, Double> {
    @Override
    public Double map(Double celsius) {
        return celsius * 9 / 5 + 32;
    }
}
DataStream<Double> f = celsiusReadings.map(new ToFahrenheit());

Named classes are easier to unit-test and avoid type-erasure problems, which is why teams often prefer them for non-trivial logic.

filter: keep or drop

filter takes a predicate and passes through only the records for which it returns true. The type does not change.

Java
DataStream<Order> bigOrders = orders.filter(o -> o.amount() >= 100);
DataStream<String> nonEmpty = lines.filter(l -> !l.isBlank());

Put filters as early as possible: records that are dropped before a keyBy never cost network, serialization or state.

flatMap: one in, any number out

flatMap gets a Collector and may emit zero, one or many records per input.

Java
DataStream<String> words = lines.flatMap((String line, Collector<String> out) -> {
    for (String w : line.split("\\s+")) {
        if (!w.isEmpty()) {
            out.collect(w);
        }
    }
}).returns(Types.STRING);

It generalises the other two: map is a flatMap that always emits once, and filter is a flatMap that emits zero or one. Use flatMap for splitting (a line into words, an order into line items), for combined parse-and-filter, and for expanding one event into several.

Stand-in: the three operators side by side

Java Stand-in: map, filter and flatMap on a list of log lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Rich functions: open, close and the runtime context

Every function interface has a rich version (RichMapFunction, RichFilterFunction, RichFlatMapFunction, ...). A rich function adds:

  • open(OpenContext): called once per parallel instance before the first record. Use it to create expensive objects: parsers, regex patterns, database connections, HTTP clients, ML models. (In Flink 1.x the signature was open(Configuration).)
  • close(): called at the end, to release resources.
  • getRuntimeContext(): access to the subtask index, the job and task name, keyed state, metrics and accumulators.
Java
public class Enrich extends RichMapFunction<Click, EnrichedClick> {
    private transient CountryLookup lookup;
    private transient Counter lookups;

    @Override
    public void open(OpenContext openContext) {
        lookup = CountryLookup.load();                        // once per subtask
        lookups = getRuntimeContext().getMetricGroup().counter("lookups");
    }

    @Override
    public EnrichedClick map(Click c) {
        lookups.inc();
        return new EnrichedClick(c, lookup.countryOf(c.ip()));
    }

    @Override
    public void close() {
        lookup.close();
    }
}

Mark such fields transient: the function object is serialized on the client and shipped to the TaskManagers, so everything non-transient must be Serializable, and a connection object is not. A frequent error is "Task not serializable" caused by a lambda that captures a non-serializable outer object.

Stand-in: lifecycle of a rich function

Java Stand-in: open once, map many times, close once

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Handling bad records

A single unparsable record that throws an exception fails the job, which restarts and reads the same record again: a crash loop. Options:

  1. Filter or catch in a flatMap: emit nothing for bad input and count it with a metric.

    Java
    lines.flatMap((String line, Collector<Event> out) -> {
        try {
            out.collect(Event.parse(line));
        } catch (IllegalArgumentException e) {
            badRecords.inc();
        }
    }).returns(Event.class);
    
  2. Route to a dead-letter stream with a side output (see Connect, Union and Side Outputs) so bad data can be inspected and replayed.

  3. Let some failures be fatal on purpose when bad data means a bug in the producer and you want to find out immediately.

Java Stand-in: parse safely with flatMap and count rejects

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which operator can emit zero, one or many records for each input?

    flatMap receives a Collector and may call collect any number of times.
  2. 2. Where should expensive setup, such as creating a database client, be done in a rich function?

    open() runs once per parallel instance on the TaskManager; transient fields are not serialized with the function.
  3. 3. Why should filters be placed early in a pipeline?

    The sooner useless records are dropped, the less work every later operator does.
  4. 4. What happens when a map function throws an exception on one bad record?

    Exceptions fail the task. Catch bad input yourself or route it to a dead-letter stream.

Summary

  • map transforms one to one, filter keeps or drops, flatMap emits any number of outputs through a Collector.
  • Use lambdas for short logic and named classes for logic you want to test or that needs explicit types.
  • Rich functions add open, close and the runtime context for setup, state and metrics; keep non-serializable members transient.
  • Guard against bad records so one input line cannot put the job into a crash loop.

Next: keyBy, Reduce and Rolling Aggregations.