Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Operations and Patterns

Patterns: Deduplication, Enrichment and CEP

Learn three recurring Flink patterns: deduplicating events with keyed state and timers, enriching streams with cached or async lookups, and detecting sequences with CEP.

Most real streaming jobs are built from a handful of recurring patterns. Learn them once and you can recognize them everywhere: removing duplicates that at-least-once delivery produces, adding reference data to events, and detecting meaningful sequences such as "three failed logins then a success". This lesson shows each in Flink and in a runnable Java stand-in.

The Flink code is read-only here, the Java programs simulate the logic. It follows Parallelism, Skew and Tuning and uses ideas from State Types, TTL and Broadcast State.

Three patterns

Deduplicate Drop events seen before.
Enrich Add reference data to events.
Detect sequences Find A then B within a time.

Each one needs bounded state.

1 / 2

Pattern 1: deduplication with keyed state

Producers retry, and at-least-once delivery means the same event can arrive twice. To remove duplicates you remember the ids you have seen. In the DataStream API that is a ValueState<Boolean> per event id, with a time-to-live so the state does not grow forever.

Java
public class Deduplicator extends KeyedProcessFunction<String, Event, Event> {
    private transient ValueState<Boolean> seen;

    @Override
    public void open(OpenContext ctx) {
        ValueStateDescriptor<Boolean> desc = new ValueStateDescriptor<>("seen", Types.BOOLEAN);
        desc.enableTimeToLive(StateTtlConfig.newBuilder(Duration.ofHours(24)).build());
        seen = getRuntimeContext().getState(desc);
    }

    @Override
    public void processElement(Event e, Context ctx, Collector<Event> out) throws Exception {
        if (seen.value() == null) {      // first time this id is seen
            seen.update(true);
            out.collect(e);
        }
    }
}
// events.keyBy(Event::id).process(new Deduplicator())

The SQL form uses ROW_NUMBER() OVER (PARTITION BY id ORDER BY ts) = 1, shown in SQL Windows, Joins and Top-N. The window of deduplication equals the TTL: a duplicate that arrives after the TTL passes through. Pick a TTL longer than the maximum retry horizon of your producers.

Java Stand-in: dedup with a time window of memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The output shows the trade-off: e1 at time 14 is emitted again because its memory has expired.

Pattern 2: enrichment

Events often carry only ids (user_id, product_id) while the analysis needs names, prices or segments. There are four common ways to add reference data, from simplest to most flexible:

  1. Static load in open(): read a small file or table once. Simple, but the data never updates.
  2. Broadcast state: stream the reference data as a second input, broadcast it to every task and keep it in broadcast state. Good for small to medium tables that change.
  3. Join in SQL: a lookup join (FOR SYSTEM_TIME AS OF proc_time) queries an external table, with a cache; a temporal join against a versioned table gives the value valid at event time.
  4. Async I/O: for each event, call a service or database without blocking, with a timeout and a capacity limit.
Java
DataStream<EnrichedOrder> enriched = AsyncDataStream.unorderedWait(
    orders,
    new AsyncFunction<Order, EnrichedOrder>() {
        @Override
        public void asyncInvoke(Order o, ResultFuture<EnrichedOrder> result) {
            customerClient.fetch(o.customerId())                 // returns a CompletableFuture
                .thenAccept(c -> result.complete(List.of(new EnrichedOrder(o, c.name()))));
        }
    },
    Duration.ofSeconds(2),        // timeout per request
    100);                         // at most 100 requests in flight

unorderedWait emits results as soon as they are ready, which gives the best throughput; orderedWait preserves input order at some cost. Put a small cache (a Caffeine cache or a MapState) in front of the service so hot keys are not fetched repeatedly, and decide what to do on a timeout (emit the event with a default, or route it to a side output for retry).

Java Stand-in: enrichment with a cache in front of a slow lookup

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Eight events needed only four database calls, and the unknown id 9 was handled with an explicit default instead of a crash.

Pattern 3: complex event processing (CEP)

Sometimes the signal is a sequence: a payment declined three times in five minutes, a temperature that rises for four consecutive readings, a user who adds to cart and does not purchase within an hour. Flink's CEP library (flink-cep) matches such patterns over keyed streams using a small state machine per key.

Java
Pattern<LoginEvent, ?> pattern = Pattern.<LoginEvent>begin("fails")
        .where(SimpleCondition.of(e -> e.type().equals("FAIL")))
        .times(3).consecutive()
    .next("success")
        .where(SimpleCondition.of(e -> e.type().equals("OK")))
    .within(Duration.ofMinutes(2));

PatternStream<LoginEvent> matches = CEP.pattern(logins.keyBy(LoginEvent::user), pattern);

DataStream<String> alerts = matches.select(m ->
    "possible brute force on " + m.get("success").get(0).user());

Key ideas: next requires the very next event to match (strict contiguity), followedBy allows other events in between, times, oneOrMore and optional set repetitions, and within bounds the time so partial matches do not live forever. The same capability exists in SQL as MATCH_RECOGNIZE.

The stand-in implements the "three failures then a success within a time limit, per user" automaton by hand, which is essentially what CEP generates for you.

Java Stand-in: three failures then a success within 120 seconds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Notice what the hand-written version already needs: per-user counters, a start time, resets and a time check. Add followedBy options, optional steps and event-time ordering and you see why CEP is worth using for real patterns.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why give dedup state a TTL?

    The TTL bounds state size and defines the deduplication horizon.
  2. 2. Which enrichment approach suits a small reference table that changes while the job runs?

    Broadcast state delivers updates to every task and keeps a local copy.
  3. 3. What does unorderedWait do in async I/O?

    Ordered wait preserves order; unordered emits as results arrive.
  4. 4. What does within(Duration) do in a CEP pattern?

    Without a time bound partial matches could live forever.

Summary

  • Deduplicate with keyed state per id and a TTL equal to your deduplication horizon (or ROW_NUMBER in SQL).
  • Enrich with static data, broadcast state, lookup or temporal joins, or async I/O with a cache and timeouts.
  • CEP and MATCH_RECOGNIZE detect event sequences with strict or relaxed contiguity and a time bound.
  • Bound every pattern's state: TTL, within, windows.

Next: Project: Sessionizing Clickstream Events