Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 37 ยท Streaming Concepts

Bounded and Unbounded Streams

Understand bounded and unbounded data, why batch is a special case of streaming, Flink's STREAMING and BATCH execution modes, and when to pick each one.

Every dataset has a shape in time. Either it has a last record, or it does not. That one property decides most of what a processing engine can and cannot do, and Flink is built around handling both with one set of ideas.

This lesson defines the two kinds of data, explains how Flink unifies them, and shows the two execution modes you can choose. It continues from The Dataflow Model and the Job Graph.

Bounded streams

A bounded stream has a defined start and end. Examples: yesterday's log files, a database table snapshot, a CSV export, a Kafka topic read from offset 0 up to a fixed end offset.

Because the end is known, a job can read everything, sort the data, build complete aggregates and then finish. Processing a bounded stream is batch processing. Results are final when the job ends, and the job can be scheduled in stages and restarted cheaply.

Unbounded streams

An unbounded stream has a start but no defined end: sensor readings, clickstream, payment events, a Kafka topic you read forever. The job has to be ready for the next event at any moment.

Because the end never comes, you cannot "wait for all the data". Questions must be rephrased: not "how many orders were there?" but "how many orders per minute?" or "what is the running total right now?". Data must be processed in the order it arrives (or by event time with watermarks, see Watermarks), and every intermediate result may later be refined.

Question Bounded Unbounded
Is there a last record? Yes No
Can you sort everything first? Yes No
When is a result final? When the job ends Never; use windows or updates
Job lifetime Finishes Runs until cancelled
Typical tools Files, tables Message queues, sockets, CDC

Batch is a special case of streaming

Older systems had one engine for batch and a different one for streams, with two APIs and two code bases (the "lambda architecture"). Flink takes the view that a bounded stream is just an unbounded stream that ends. The same operators work for both. Since Flink 2.0 the old, separate DataSet API is gone entirely: for batch you use the same DataStream, Table API or SQL programs.

The difference that remains is how the scheduler may run the job, which is what execution modes control.

Runtime execution modes

Set the mode on the environment (or with -Dexecution.runtime-mode=BATCH on the command line):

Java
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setRuntimeMode(RuntimeExecutionMode.BATCH);        // or STREAMING, or AUTOMATIC
  • STREAMING (the default). All tasks run at the same time, data flows continuously, state is kept in a state backend and checkpoints protect it. Works for bounded and unbounded sources.
  • BATCH. Only for bounded sources. The scheduler can run the job in stages, writing intermediate results to disk, sorting by key and processing one key at a time. It uses fewer resources and gives final results faster, but it does not use checkpoints; failures re-run the affected stages instead.
  • AUTOMATIC. Flink picks BATCH if all sources are bounded, otherwise STREAMING.

Differences you will notice in BATCH mode:

  • Results of a rolling sum are emitted once per key at the end, not on every record.
  • Event time still works, but the watermark jumps to the maximum at the end of the input, so windows fire when all data has been read. Records are never late.
  • Only the batch-friendly state handling is used, so processing keyed data needs no state backend.
  • Checkpointing is not available, and some operators (for example those requiring continuous output) are not allowed.

Stand-in: the same logic in two modes

The stand-in below counts words in a bounded list in two different ways that mirror the modes: STREAMING (emit an update per record) and BATCH (sort by key, then emit one result per key).

Java Stand-in: STREAMING versus BATCH output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Both reach the same final counts. BATCH emits three lines and holds only one key's counter at a time; STREAMING emits six lines and keeps every key's state alive.

Reading a file as a bounded stream

The FileSource is bounded by default. To make it unbounded, you enable continuous monitoring of the directory:

Java
FileSource<String> bounded = FileSource
    .forRecordStreamFormat(new TextLineInputFormat(), new Path("/data/logs"))
    .build();                                           // reads the files once, then ends

FileSource<String> unbounded = FileSource
    .forRecordStreamFormat(new TextLineInputFormat(), new Path("/data/logs"))
    .monitorContinuously(Duration.ofSeconds(30))
    .build();                                           // keeps looking for new files forever

KafkaSource works the same way: setBounded(OffsetsInitializer.latest()) turns a topic read into a bounded job, which is handy for backfills and tests.

Where each shape fits

  • Backfills and reprocessing. Run the same program in BATCH mode over historical data after fixing a bug.
  • Daily reports. A bounded source plus BATCH mode: cheap, and finishes.
  • Alerts and dashboards. Unbounded plus STREAMING.
  • Tests. Use a small bounded source (fromData) in STREAMING mode, so the same code path as production runs.
Java Stand-in: a bounded read can finish, an unbounded one cannot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these is a bounded stream?

    A file has a last record, so reading it is bounded.
  2. 2. Which runtime execution mode can only be used when all sources are bounded?

    BATCH needs a known end so it can run in stages and sort data. STREAMING works for both.
  3. 3. In BATCH mode, when does a rolling sum per key emit results?

    BATCH mode groups data per key and emits the final result, not an update per record.
  4. 4. How does Flink view the relation between batch and streaming?

    A bounded stream is an unbounded stream that ends; one set of operators serves both.

Summary

  • Bounded data has an end, so jobs can sort, finish and give final answers; unbounded data never ends and needs windows or running results.
  • Flink runs both with the same API; the DataSet API no longer exists in Flink 2.x.
  • STREAMING mode runs everything continuously with state and checkpoints; BATCH mode runs in stages for bounded input only; AUTOMATIC chooses.
  • Sources like FileSource and KafkaSource can be made bounded or unbounded.

Next: Stateless and Stateful Processing.