Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 37 ยท Projects and Practice

Practice: Flink Review Exercises

Test what you learned about Flink state, checkpoints, SQL, connectors and operations with worked exercises, runnable solutions, design scenarios and a review quiz.

You have covered state, checkpoints, savepoints, the Table API and SQL, connectors, deployment and operations. This last lesson is practice. It has four exercises with runnable solutions (Java and SQL stand-ins, since the site cannot run Flink), design scenarios to answer on paper and a review quiz. Try each exercise yourself before you open the solution.

Concepts come from the whole chapter range; follow the links if something feels rusty: Keyed State in Flink, Checkpoints and Barriers, Dynamic Tables and Changelogs, Exactly-Once Sinks and Two-Phase Commit and Monitoring and Backpressure.

Exercise 1: which windows fire?

A job uses 10-second tumbling event-time windows and a bounded-out-of-orderness watermark of 5 seconds (watermark = highest timestamp seen minus 5 seconds). Events arrive with these timestamps in seconds, in this order: 3, 8, 12, 7, 16, 21, 4, 27. Decide, after each event, which windows have fired, and which events are late (dropped because their window already fired).

Think first about two rules: a window [s, s+10) fires when the watermark reaches s+10 or more, and an event whose window has already fired is late.

Java Solution 1: watermarks, firing and late events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The event with timestamp 4 arrives after window [0,10) fired (the watermark reached 11 when 16 arrived), so it is late. The event 7 arrived before that and is included, which shows how the 5 seconds of out-of-orderness absorbs mild disorder but not all of it. How would you keep 4? Raise the bound, use allowedLateness, or send late data to a side output.

Exercise 2: size a job

A job must process 120,000 events per second at peak. In a load test one parallel instance of the heaviest operator handled 15,000 events per second at 70 percent CPU. You want a 30 percent headroom target, 4 slots per TaskManager and checkpoints that take less than a third of a 60-second interval. Compute the parallelism, the number of TaskManagers and check the checkpoint budget when state is 24 GB, with incremental checkpoints uploading about 1.5 GB and storage that accepts 200 MB per second in total.

Java Solution 2: capacity arithmetic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Parallelism of 8 (two TaskManagers) meets the throughput; the arithmetic also shows why a 24 GB state needs incremental checkpoints to fit the budget. Real numbers will differ, so confirm with a test that includes checkpoints, skew and a realistic key distribution.

Exercise 3: SQL, the latest status per order

An order_events table has one row for each status change. Return the latest status of every order, and the number of orders per current status. In Flink SQL this is the deduplication pattern with ROW_NUMBER() ... ORDER BY ts DESC followed by a GROUP BY; the result is an updating table. Write the query yourself, then run the portable version.

SQL Solution 3: latest status per order, then counts

Press Run (or Ctrl + Enter) to see the result here.

In Flink the counts would change over time: when order 2 moves from paid to shipped, the paid count is retracted by one and shipped grows by one. That chain of retractions is precisely the changelog you studied earlier, and it is why the sink needs an upsert key such as status.

Exercise 4: design scenarios

Answer these in a few sentences each. A suggested answer follows each question.

  1. A job restarts every hour with java.io.NotSerializableException. What is likely wrong? A function class holds a non-serializable field (a client or connection) initialized in the constructor; create it in open() and mark it transient.
  2. After a new release, the job starts with an empty state. The operators had no .uid(...), so the new job graph produced different ids, or the job was started without -s <savepoint>. Set stable uids and restore from the savepoint.
  3. Source lag grows, the sink operator is idle and the middle operator is 100 percent busy. The middle operator is the bottleneck. Check skew first, then its parallelism and per-record cost.
  4. Users see duplicate Kafka messages from your job although it uses exactly-once. Consumers probably use read_uncommitted; set isolation.level=read_committed.
  5. A temporal join suddenly returns nothing. The versioned table's watermark is not advancing (an idle partition or no updates), so the join waits; configure idleness or a source watermark.
  6. Checkpoints take 40 s and sometimes time out under load. Backpressure delays barriers. Fix the bottleneck, then enable unaligned checkpoints or buffer debloating, and use incremental RocksDB checkpoints.

Review quiz

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A ValueState handle works inside which functions?

    Keyed state is scoped to the current key that keyBy establishes.
  2. 2. What must a sink do to deliver end-to-end exactly-once when it cannot upsert?

    Two-phase commit ties visibility to checkpoint completion.
  3. 3. Which statement about savepoints is true?

    Savepoints are created on demand and kept until you remove them.
  4. 4. Which SQL query shape produces an append-only result?

    A closed window emits a final row and drops its state.
  5. 5. Which deployment mode isolates one job from another best?

    Each application gets its own cluster.

Where to go next

Build one of the projects against a real cluster: run Flink locally in standalone mode, read from a Kafka topic in Docker and write to a PostgreSQL table. Read the Flink documentation pages for your connectors, set up a metrics dashboard and practise a stop-with-savepoint upgrade. The skills transfer: state, time and failure are the three hard problems of every stream processor.

Summary

  • Watermarks decide when windows fire; the out-of-orderness bound trades latency for completeness, and truly late data needs lateness or a side output.
  • Capacity planning is arithmetic on measured per-instance throughput, headroom and checkpoint budget.
  • The latest-state-per-key pattern is ROW_NUMBER ... DESC plus a changelog-aware sink.
  • Most production issues come from missing uids, non-serializable functions, skew, backpressure and consumer isolation settings.

Next: back to the Apache Flink track