Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 37 ยท Getting Started

What Is Apache Flink?

Learn what Apache Flink is, why stream processing exists, how it differs from batch tools like Spark, and where companies use it for real-time analytics and pipelines.

Most data is born as a stream. A card is swiped, a sensor reports a temperature, a user taps a button, a server writes a log line. Each of these is an event, and events keep arriving forever. For years the standard way to deal with them was to collect them into files or tables and run a job every hour or every night. That works, but the answer you get is always old.

Apache Flink is an open-source framework and distributed engine for processing data streams as they arrive, with results in milliseconds to seconds, and with strong guarantees about correctness. It also runs ordinary batch jobs on files, because Flink treats a finite file as just a stream that happens to end.

About the examples. Flink is a cluster framework that needs a Java runtime, a job submission step and usually Kafka or another system. The site cannot run it in your browser. Flink code in these lessons is shown read-only, and the runnable blocks are small, clearly labelled Java stand-ins that reproduce the underlying idea (grouping events, counting per key, windows) so you can experiment with it.

Stream processing in one picture

A streaming job is a pipeline: events flow from a source, pass through operators that transform them, and end in a sink that stores or forwards the result.

flowchart LR
    S["Source<br/>Kafka, files, sockets"] --> M["Operators<br/>map, filter, keyBy, window"]
    M --> K["Sink<br/>database, Kafka, dashboard"]

A Flink job is a graph of sources, operators and sinks. Events keep flowing through it.

Unlike a batch job, a streaming job never finishes by itself. It runs until you stop it, and it must survive machine failures, restarts and upgrades while holding on to what it has learned so far (running counts, open windows, user sessions). That remembered information is called state, and managing it well is what Flink is famous for.

  • True streaming. Events are processed one by one (in small network buffers) rather than collected into micro-batches first, so latency can be very low.
  • Event-time semantics. Flink can order results by when events happened, not when they arrived, even if the network delivers them late or out of order. See Event Time and Processing Time.
  • Large, fault-tolerant state. Per-key state lives next to the computation, is checkpointed to durable storage and is restored after a failure.
  • Exactly-once state consistency. After a crash, Flink resumes from a consistent snapshot, so counters are not double-counted. See Delivery Guarantees.
  • Several APIs for different needs. The DataStream API gives full control, the Table API and SQL give a declarative style, and the lowest level, ProcessFunction, gives you timers and state directly.
  • Batch and streaming in one engine. The same SQL or DataStream program can run in a streaming or batch execution mode.
Use case Example
Real-time analytics Live dashboards: orders per minute, active users now
Event-driven applications Fraud detection, alerting, rule engines reacting per event
Data pipelines (streaming ETL) Clean, enrich and move data from Kafka into a warehouse or lake
Change data capture Keep a search index in sync with a database's change log
Machine-learning features Compute fresh per-user features for online models
  • Apache Spark started as a batch engine and processes streams in small batches (Structured Streaming also has a continuous mode). It is excellent for large analytics and machine learning. Flink was designed stream-first, so it usually offers lower latency and richer time and state handling. See the Apache Spark tutorial for the other side.
  • Apache Kafka is the storage and transport layer for event logs, with the lightweight Kafka Streams library on top. Flink is a separate processing cluster that very often reads from and writes to Kafka. See Kafka.
  • A plain Java service can process events too, but you would write the partitioning, the state storage, the recovery and the time handling yourself. Flink provides all of it.

The idea behind it, runnable

The smallest streaming idea is "update a result every time an event arrives". The stand-in below counts page views per page, one event at a time, and prints the running result after each event, which is exactly what a stateful streaming operator does.

Java Stand-in: a running count updated per event

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

A batch job would print only the last line, after reading everything. A stream job prints a fresh answer after every event. In real Flink the map lives in managed, checkpointed keyed state, and the loop is a distributed operator running in parallel.

For comparison, the same idea as a batch of events grouped at the end:

Java Stand-in: batch result versus streaming results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Both give the same final numbers. The difference is when you can look at them, and that streaming has to keep the running map correct forever, even when machines fail.

This is the shape of a real Flink job (read-only). Do not worry about the details yet; the next lessons explain each line.

Java
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

env.fromData("/home", "/pricing", "/home")
   .map(page -> Tuple2.of(page, 1))
   .returns(Types.TUPLE(Types.STRING, Types.INT))
   .keyBy(t -> t.f0)
   .sum(1)
   .print();

env.execute("page views");

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main difference between a batch job and a streaming job?

    Batch input is finite so the job finishes; streaming input never ends, so the job keeps running and keeps its state.
  2. 2. Which statement about Flink and batch processing is true?

    Batch is a special case of streaming in Flink: the input is bounded, so the job can finish.
  3. 3. What is 'state' in a streaming job?

    State is the remembered information (counts, open windows, sessions) that Flink stores and checkpoints so it survives failures.
  4. 4. Which is a typical Flink use case?

    Reacting to events within milliseconds, with state per card, is a core streaming use case.

Summary

  • Flink is a distributed engine for stateful processing of unbounded (and bounded) data streams with low latency.
  • A job is a graph of sources, operators and sinks that runs until stopped.
  • Its strengths are event time, large fault-tolerant state and exactly-once state consistency.
  • Spark, Kafka and plain services overlap in some areas but solve different parts of the problem.
  • The runnable examples in this track are stand-ins; real Flink code is shown read-only.

Next: see how a cluster runs your job in Flink Architecture.