Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Projects and Practice

Project: Real-Time Fraud Alerts

Build a fraud alert pipeline: velocity, amount-spike and impossible-travel rules with keyed state, timers and broadcast rules, with Flink code and runnable Java and SQL stand-ins.

Fraud detection is the textbook streaming use case: a decision must be made in milliseconds, using history about the account (what is normal for this customer?) and a set of rules that analysts change often. In this project you design a pipeline with three rules, implement each as keyed state logic, make the thresholds updatable at run time with broadcast state, and backtest a rule with SQL.

The Flink code below is real and read-only (the site cannot run Flink); the programs you can run are Java and SQL stand-ins on a fixed data set. It builds on State Types, TTL and Broadcast State, Patterns: Deduplication, Enrichment and CEP and the previous project, Sessionizing Clickstream Events.

Requirements

Input: card transactions {tx_id, account, amount, country, ts} (seconds). Output: alerts {tx_id, account, rule, detail}. Rules:

  1. Velocity: more than 3 transactions by one account within 60 seconds.
  2. Amount spike: an amount above 5 times the account's running average (after at least 3 previous transactions).
  3. Impossible travel: two transactions from different countries within 10 minutes.

Thresholds (the 3, the 5, the 10 minutes) must be changeable without redeploying the job.

flowchart LR
    T["Transactions<br/>Kafka"] --> D["Dedup by tx_id"]
    D --> K["keyBy account"]
    R["Rule updates<br/>Kafka"] -- broadcast --> P
    K --> P["Fraud detector<br/>keyed + broadcast state"]
    P --> A["Alerts topic"]
    P -. "late/invalid" .-> X["Side output"]

One detector function sees each transaction, the account's state and the broadcast thresholds.

One KeyedBroadcastProcessFunction holds the three rules. Per account it keeps: a ListState<Long> of recent timestamps (for velocity), a running count and sum (for the average), and the last country with its timestamp.

Java
public class FraudDetector
        extends KeyedBroadcastProcessFunction<String, Tx, Thresholds, Alert> {

    private transient ListState<Long> recent;          // timestamps in the last 60 s
    private transient ValueState<double[]> stats;      // {count, sum}
    private transient ValueState<Tx> lastTx;

    @Override
    public void processElement(Tx tx, ReadOnlyContext ctx, Collector<Alert> out) throws Exception {
        Thresholds th = ctx.getBroadcastState(THRESHOLDS).get("current");
        if (th == null) { th = Thresholds.defaults(); }

        // 1) velocity: keep only timestamps inside the window, then count
        List<Long> window = new ArrayList<>();
        for (Long t : recent.get()) { if (tx.ts() - t < th.velocityWindowSec()) window.add(t); }
        window.add(tx.ts());
        recent.update(window);
        if (window.size() > th.maxTxPerWindow()) out.collect(Alert.of(tx, "VELOCITY", window.size() + " tx in window"));

        // 2) amount spike versus the running average
        double[] s = stats.value() == null ? new double[]{0, 0} : stats.value();
        if (s[0] >= 3 && tx.amount() > th.spikeFactor() * (s[1] / s[0])) {
            out.collect(Alert.of(tx, "SPIKE", tx.amount() + " vs avg " + (s[1] / s[0])));
        }
        stats.update(new double[]{s[0] + 1, s[1] + tx.amount()});

        // 3) impossible travel
        Tx prev = lastTx.value();
        if (prev != null && !prev.country().equals(tx.country())
                && tx.ts() - prev.ts() < th.travelWindowSec()) {
            out.collect(Alert.of(tx, "TRAVEL", prev.country() + " -> " + tx.country()));
        }
        lastTx.update(tx);
    }

    @Override
    public void processBroadcastElement(Thresholds t, Context ctx, Collector<Alert> out) throws Exception {
        ctx.getBroadcastState(THRESHOLDS).put("current", t);
    }
}

Design choices worth noticing: only small aggregates are kept per account (no full history), the velocity list is pruned on every event so it stays bounded, and the thresholds live in broadcast state so an analyst's change reaches all parallel instances (see the broadcast section of State Types, TTL and Broadcast State). To make idle accounts free their state, enable TTL on the descriptors or register a timer that clears state after, say, 24 hours of silence.

Step 2: the logic as a runnable simulation

The Java program below implements the same three rules with a map per account instead of Flink state, so you can feed it transactions and read the alerts. The data includes one example of each rule.

Java Stand-in: three fraud rules over per-account state

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The velocity alert fires for the fourth transaction of account A, the spike for B's 400 (versus an average of about 15), and the travel alert for C moving from France to Japan in under ten minutes.

Step 3: changing thresholds while running

In production a risk analyst edits a rules table; a CDC connector or a Kafka topic publishes the new Thresholds object, and processBroadcastElement stores it. Every parallel instance applies it to the next event. Two cautions: the main stream and the broadcast stream are not ordered relative to each other, so an event near the change can use the old or the new value, and the broadcast state must be small, because it is replicated to every task and included in every checkpoint.

Step 4: backtesting a rule in SQL

Before enabling a new rule in the live job, analysts replay stored history through the same logic, often in SQL. The velocity rule is a counting condition over a trailing time range. This query counts, for each transaction, how many transactions the same account made in the 60 seconds up to and including it.

SQL Backtesting the velocity rule (portable SQL)

Press Run (or Ctrl + Enter) to see the result here.

Counting rows per transaction shows that t4 is the first to exceed three. In Flink SQL the same condition is an OVER window with a RANGE BETWEEN INTERVAL '60' SECOND PRECEDING AND CURRENT ROW frame, which is computed incrementally and allows the rule to run in the same SQL job.

Step 5: production notes

  • Exactly-once alerts: write alerts to a transactional Kafka sink (see Exactly-Once Sinks and Two-Phase Commit) or make downstream handling idempotent using tx_id plus rule as a key.
  • Latency budget: with a transactional sink, alerts appear at checkpoint completion; if you need sub-second alerts, use an at-least-once sink and deduplicate downstream.
  • Hot accounts: merchants or test accounts with huge volume cause skew; consider keying by card instead of merchant (Parallelism, Skew and Tuning).
  • Auditing: send every decision, with the thresholds used, to a log topic so decisions are reproducible.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why prune the velocity list on every event?

    Old timestamps are useless once outside the window, and unbounded lists would leak memory.
  2. 2. How do updated thresholds reach all parallel instances?

    processBroadcastElement runs on every instance and stores the new values.
  3. 3. What is a caveat of broadcast state?

    Events near a rule change may see either value, and the whole map is copied to every task.
  4. 4. Why use only running aggregates (count and sum) for the spike rule?

    Count and sum give the running average in constant space.

Summary

  • A fraud detector is keyed by account and keeps small, bounded state: recent timestamps, running aggregates and the last transaction.
  • Broadcast state makes thresholds changeable at run time; mind ordering and size.
  • Rules such as velocity, spike and impossible travel map directly to state logic; SQL backtests validate them on history.
  • Choose the sink guarantee by your latency budget and make alert handling idempotent.

Next: Project: Real-Time Leaderboard