Skip to content
Tutorials Center
My learning

Topics

Apache Flink Tutorial

Learn Apache Flink step by step: 37 lessons in 9 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Getting Started

  1. 1 What Is Apache Flink? Learn what Apache Flink is, why stream processing exists, how it differs from batch tools like Spark, and where companies use it for real-time analytics and pipelines.
  2. 2 Flink Architecture Understand how a Flink cluster runs a job: client, JobManager, Dispatcher, ResourceManager, TaskManagers, task slots, parallelism and how work is spread over machines.
  3. 3 Setting Up Flink and Running Your First Job Install Apache Flink, start a local cluster, create a Maven project and run a streaming WordCount job. Includes the web UI tour and a runnable stand-in for the logic.
  4. 4 The Dataflow Model and the Job Graph Learn how a Flink program becomes a dataflow graph: sources, transformations, sinks, operator chaining, forward and hash exchanges, and how to read the plan.

Streaming Concepts

  1. 5 Bounded and Unbounded Streams Understand bounded and unbounded data, why batch is a special case of streaming, Flink's STREAMING and BATCH execution modes, and when to pick each one.
  2. 6 Stateless and Stateful Processing Learn the difference between stateless and stateful operators in Flink, what keyed state and operator state are, why state matters, and how it stays safe in failures.
  3. 7 Parallelism, keyBy and Key Groups See how Flink spreads work with parallelism, routes records by key with keyBy, and uses key groups and max parallelism so keyed state can be rescaled safely.
  4. 8 Delivery Guarantees and Exactly-Once Learn at-most-once, at-least-once and exactly-once guarantees, how Flink checkpoints and barriers give exactly-once state, and why end-to-end needs idempotent or transactional sinks.
  5. 9 Backpressure and Flow Control Understand backpressure in Flink: how credit-based flow control slows fast producers, how to spot the bottleneck in the web UI, and the usual fixes for slow sinks and skew.

DataStream API

  1. 10 DataStream Basics: Environment, Sources and Sinks Learn the structure of every Flink DataStream program: the execution environment, creating sources, data types and TypeInformation, sinks, and running with execute.
  2. 11 Map, Filter and FlatMap Learn Flink's basic stateless transformations: map, filter, flatMap, lambdas versus function classes, rich functions with open and close, and how to handle bad records.
  3. 12 keyBy, Reduce and Rolling Aggregations Group streams by key and aggregate them in Flink: keyBy selectors, rolling sum, min, max, minBy and maxBy, reduce, and how to pick keys safely, with runnable examples.
  4. 13 Process Functions: State and Timers Use Flink's low-level ProcessFunction: access records, keyed state and timers in one place to build inactivity alerts, deduplication and custom logic that built-in operators cannot do.
  5. 14 Connect, Union and Side Outputs Combine and split streams in Flink: union for same-type streams, connect with CoMapFunction for different types, broadcast streams for rules, and side outputs for late data and dead letters.

Time and Windows

  1. 15 Event Time and Processing Time Understand the notions of time in Flink: event time, processing time and ingestion time, why late and out-of-order events break wall-clock results, and how to assign timestamps.
  2. 16 Watermarks Learn how Flink watermarks measure progress of event time, how forBoundedOutOfOrderness works, how watermarks combine across inputs, and how to handle idle sources.
  3. 17 Tumbling and Sliding Windows Aggregate unbounded streams in time slices: Flink tumbling and sliding windows, window assigners, offsets, reduce, aggregate and ProcessWindowFunction, with runnable simulations.
  4. 18 Session Windows, Global Windows and Triggers Group events by activity with Flink session windows and gap detection, build custom windows with GlobalWindows, and control firing with triggers, evictors and window joins.
  5. 19 Late Data and Allowed Lateness Handle late events in Flink event-time windows: what makes an event late, allowedLateness and window updates, side outputs for dropped events, and choosing between delay and lateness.

State and Fault Tolerance

  1. 20 Keyed State in Flink Learn why stream processing needs state, how Flink scopes state to the current key with ValueState, and how to model it with runnable Java stand-ins.
  2. 21 State Types, TTL and Broadcast State Choose between ValueState, ListState, MapState and reducing state, expire old entries with state TTL, and share rules to all tasks with broadcast state in Flink.
  3. 22 Checkpoints and Barriers Understand how Flink checkpoints work: barriers flow with the data, operators snapshot their state, sources remember offsets, and a failure rewinds the whole job consistently.
  4. 23 State Backends and Savepoints Pick between the HashMap and RocksDB state backends, enable incremental checkpoints, and use savepoints with stable operator uids to upgrade a Flink job safely.

Table API and SQL

  1. 24 Table API First Steps Start with the Flink Table API: create a TableEnvironment, define tables with connectors, write relational queries, and convert between Table and DataStream.
  2. 25 Flink SQL Queries Write Flink SQL: create tables with connectors, run SELECT, WHERE, GROUP BY, CASE and CTE queries, use INSERT INTO, the SQL client and statement sets, with runnable portable SQL.
  3. 26 Dynamic Tables and Changelogs Learn how Flink turns streams into dynamic tables and back: append-only versus updating results, changelog row kinds, retractions and upserts, with runnable Java stand-ins.
  4. 27 SQL Windows, Joins and Top-N Use Flink SQL window table functions (TUMBLE, HOP, CUMULATE), interval and temporal joins, Top-N and deduplication patterns, with runnable portable SQL equivalents.

Connectors and Deployment

  1. 28 Connectors: Sources and Sinks Understand Flink connectors: the unified Source API with splits and enumerators, KafkaSource and KafkaSink, FileSink, JDBC and CDC, and offsets that make replay possible.
  2. 29 Exactly-Once Sinks and Two-Phase Commit See how Flink achieves end-to-end exactly-once: replayable sources, checkpointed state and transactional or idempotent sinks with a two-phase commit protocol, with runnable simulations.
  3. 30 Deployment Modes and Clusters Learn the Flink cluster architecture (JobManager, TaskManagers, slots), operator chaining, application and session modes, Kubernetes and YARN, and high availability.

Operations and Patterns

  1. 31 Monitoring and Backpressure Monitor Flink jobs with the web UI and metrics, read busy, idle and backpressured time, find the bottleneck operator, and set alerts for checkpoints, lag and restarts.
  2. 32 Parallelism, Skew and Tuning Understand Flink parallelism and key groups, detect data skew, fix hot keys with two-phase (salted) aggregation, and tune buffers, state and checkpoints for throughput.
  3. 33 Patterns: Deduplication, Enrichment and CEP Learn three recurring Flink patterns: deduplicating events with keyed state and timers, enriching streams with cached or async lookups, and detecting sequences with CEP.

Projects and Practice

  1. 34 Project: Sessionizing Clickstream Events Build a clickstream sessionization pipeline: group events into sessions with an inactivity gap, handle late events, compute session metrics, and see the Flink code and runnable stand-ins.
  2. 35 Project: Real-Time Fraud Alerts Build a fraud alert pipeline: velocity, amount-spike and impossible-travel rules with keyed state, timers and broadcast rules, with Flink code and runnable Java and SQL stand-ins.
  3. 36 Project: Real-Time Leaderboard Build a real-time game leaderboard in Flink: running scores, a global Top-N with changelog output, windowed daily boards, and an upsert sink, with runnable stand-ins.
  4. 37 Practice: Flink Review Exercises Test what you learned about Flink state, checkpoints, SQL, connectors and operations with worked exercises, runnable solutions, design scenarios and a review quiz.