Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 37 ยท DataStream API

Connect, Union and Side Outputs

Combine and split streams in Flink: union for same-type streams, connect with CoMapFunction for different types, broadcast streams for rules, and side outputs for late data and dead letters.

Real pipelines are not single straight lines. You merge two Kafka topics, enrich events with a stream of rules, or send bad records somewhere else. Flink has a small set of operators for combining and splitting streams, and they differ in one key way: whether the streams have the same type and whether they share state.

This lesson covers union, connect, broadcast state and side outputs. It follows Process Functions.

union: merge streams of the same type

union interleaves any number of streams of identical type into one. Records are not reordered or combined; they just arrive in whatever order the inputs deliver them.

Java
DataStream<Event> web = ...;
DataStream<Event> mobile = ...;
DataStream<Event> iot = ...;

DataStream<Event> all = web.union(mobile, iot);

Use it to treat several sources of the same shape as one. Note that a stream can be unioned with itself, which duplicates every record (rarely useful, sometimes handy to test).

connect: two streams, two types, shared state

connect joins two streams whose types may differ. The result is a ConnectedStreams<A, B>; you handle each side with its own method but in one function instance, so both sides can read and write the same state.

Java
DataStream<Order> orders = ...;
DataStream<Payment> payments = ...;

orders.connect(payments)
      .keyBy(Order::orderId, Payment::orderId)         // key both by the same value
      .process(new KeyedCoProcessFunction<String, Order, Payment, String>() {
          private transient ValueState<Order> pendingOrder;

          @Override
          public void open(OpenContext c) {
              pendingOrder = getRuntimeContext().getState(new ValueStateDescriptor<>("order", Order.class));
          }

          @Override
          public void processElement1(Order o, Context ctx, Collector<String> out) throws Exception {
              pendingOrder.update(o);                    // wait for the payment
          }

          @Override
          public void processElement2(Payment p, Context ctx, Collector<String> out) throws Exception {
              Order o = pendingOrder.value();
              if (o != null) {
                  out.collect("order " + o.orderId() + " paid " + p.amount());
                  pendingOrder.clear();
              }
          }
      });

Without keyBy, the stateless versions are CoMapFunction (map1, map2) and CoFlatMapFunction:

Java
orders.connect(payments).map(new CoMapFunction<Order, Payment, String>() {
    public String map1(Order o)   { return "order " + o.orderId(); }
    public String map2(Payment p) { return "payment " + p.orderId(); }
});

The difference from union: types can differ, and the combined operator can keep state that both streams touch. A hand-written join of this kind is a stateful stream join; Flink also has the interval join and window join, covered in later lessons.

Stand-in: union versus connect

Java Stand-in: union merges, connect correlates through shared state

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Because both sides can arrive first, each side stores what it has and checks the other side's state. This is the same trick Flink's own joins use, together with timers to expire entries that never match.

Broadcast state: rules to every subtask

Sometimes one small stream (rules, a config, a blocklist) must reach every parallel instance of the main stream's operator. Use a broadcast stream:

Java
MapStateDescriptor<String, Rule> rulesDescriptor =
    new MapStateDescriptor<>("rules", Types.STRING, Types.POJO(Rule.class));

BroadcastStream<Rule> rules = ruleUpdates.broadcast(rulesDescriptor);

transactions
    .keyBy(Tx::cardId)
    .connect(rules)
    .process(new KeyedBroadcastProcessFunction<String, Tx, Rule, Alert>() {
        @Override
        public void processElement(Tx tx, ReadOnlyContext ctx, Collector<Alert> out) throws Exception {
            for (var entry : ctx.getBroadcastState(rulesDescriptor).immutableEntries()) {
                if (entry.getValue().matches(tx)) {
                    out.collect(new Alert(tx, entry.getKey()));
                }
            }
        }

        @Override
        public void processBroadcastElement(Rule rule, Context ctx, Collector<Alert> out) throws Exception {
            ctx.getBroadcastState(rulesDescriptor).put(rule.name(), rule);   // same update on every subtask
        }
    });

Every subtask receives every rule and keeps its own identical copy in broadcast state. The main side only gets read-only access to it, so all copies stay equal. This pattern powers dynamic rule engines that change without redeploying the job.

Side outputs: more than one output

An operator normally has one output stream. A side output is an extra, typed output you can emit alongside it from a process function. Two typical uses: late data from windows (see Late Data and Allowed Lateness) and dead-letter streams for bad records.

Java
final OutputTag<String> badRecords = new OutputTag<String>("bad-records") {};   // anonymous subclass keeps the type

SingleOutputStreamOperator<Reading> parsed = lines.process(new ProcessFunction<String, Reading>() {
    @Override
    public void processElement(String line, Context ctx, Collector<Reading> out) {
        try {
            out.collect(Reading.parse(line));          // main output
        } catch (Exception e) {
            ctx.output(badRecords, line);              // side output
        }
    }
});

DataStream<String> rejected = parsed.getSideOutput(badRecords);
rejected.sinkTo(deadLetterSink);

Remember the empty {} after new OutputTag<...>("id"): it creates an anonymous subclass so Flink can read the generic type at runtime.

Java Stand-in: split valid records and a dead-letter stream

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The earlier filter-only approach in Map, Filter and FlatMap simply counted the rejects. Here you keep the bad records, so they can be stored, inspected and replayed after a fix.

Which one do I need?

Goal Operator
Merge streams of the same type union
Correlate two streams of different types with shared state connect + (Keyed)CoProcessFunction
Give a small, changing stream to all subtasks broadcast + BroadcastProcessFunction
Emit additional streams (late data, errors) side outputs with OutputTag
Choose between two streams after a condition side output, or two filter calls (the old split is removed)

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which operator can combine two streams of different types?

    union requires identical types. connect produces ConnectedStreams with two input methods.
  2. 2. Why do two connected streams often use keyBy on both sides?

    Keyed connected streams share per-key state, which is how correlation works.
  3. 3. How does the main stream see broadcast state?

    Only the broadcast side may modify the state; the keyed side reads it.
  4. 4. What is a common use of side outputs?

    Side outputs allow extra typed output streams, such as dead letters or late data.

Summary

  • union merges same-type streams; connect links two streams of different types so one function can handle both with shared state.
  • Broadcast streams distribute a small, changing stream (rules) to every subtask via read-only broadcast state.
  • Side outputs add typed extra outputs from process functions: dead letters, late data, routing.
  • Keyed connect plus state is the basis of hand-written stream joins.

Next, time: Event Time and Processing Time.