Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 47 ยท Data Ingestion and Processing Models

Batch vs Stream Processing

Understand bounded and unbounded data, latency versus throughput, event time, tumbling and sliding windows, watermarks, and Lambda and Kappa architectures.

Two questions decide how you process data: how much of it do you have at once, and how fast do you need an answer? If you have a finished file and can wait an hour, you run a batch job. If events keep arriving and you need the answer within seconds, you run a stream job. The two models look similar on a whiteboard, but they differ in how they treat time, failures and results.

This lesson builds the vocabulary: bounded and unbounded data, event time versus processing time, windows, watermarks and late events, and the Lambda and Kappa architectures. It follows Getting Data In.

Streaming engines cannot run on the site. The runnable examples are Python stand-ins that implement windowing and watermarks by hand, so you can see every decision the engine makes.

Bounded and unbounded data

A bounded data set has a beginning and an end: yesterday's logs, a table snapshot. A job reads all of it, computes, and finishes. MapReduce is the classic batch model.

An unbounded data set never ends: clickstreams, sensor readings, payment events. A job over it never "finishes"; it keeps producing results as data flows in. You cannot wait for the end before computing, so streaming systems must decide when a result is ready.

Batch or stream?

Batch Bounded input, one result at the end.
Stream Unbounded input, results keep coming.

Batch is a special case of stream with an end.

1 / 2

Latency versus throughput. Batch jobs can read large blocks sequentially, compress well and use every disk, so they reach high throughput, but the result arrives after the job. Streaming handles records one at a time or in tiny micro-batches, which costs more per record but gives answers in milliseconds to seconds. Many pipelines start as batch and get a streaming layer only where freshness truly pays for the extra complexity.

Two kinds of time

Every event has at least two timestamps:

  • Event time: when it actually happened (the moment a user clicked, according to the device clock).
  • Processing time: when your system saw it.

In a perfect world they are equal. In reality events arrive late and out of order: a phone loses signal in a tunnel and uploads ten minutes of clicks later; a network partition delays a whole shard. If you count "clicks per minute" by processing time, those clicks land in the wrong minute and your numbers depend on network luck. For correct results, aggregate by event time.

Windows

You cannot sum an infinite stream, so you cut it into windows and aggregate each one.

  • Tumbling window: fixed size, no overlap. "Every 60 seconds": 0-60, 60-120, ... Each event belongs to exactly one window.
  • Sliding (hopping) window: fixed size, advancing by a smaller step. "Last 60 seconds, updated every 20 seconds." Each event belongs to several windows.
  • Session window: closes after a gap of inactivity (for example 30 minutes without a click per user). Window length varies per key.

In SQL-ish pseudo-notation (read-only):

TEXT
SELECT window_start, COUNT(*)
FROM clicks
GROUP BY TUMBLE(event_time, INTERVAL '1' MINUTE);

Watermarks and late events

If events can be late, when may a window be considered complete? A watermark is the engine's moving claim: "I believe I have seen all events with event time up to T." It is usually computed as the largest event time seen so far minus an allowed lateness. When the watermark passes the end of a window, the window fires and emits its result.

Events that arrive after their window has fired are late. The system can drop them, send them to a side output, or update the earlier result (allowed lateness), trading completeness for latency. There is no free choice: a long delay gives more complete results but later answers.

The first stand-in processes a small stream in arrival order. It counts clicks per tumbling 10-second event-time window, and fires a window when the watermark passes its end, with a lateness allowance of 5 seconds.

Python Tumbling windows with a watermark and late events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Follow the printed lines. The event at time 9 arrives out of order but before its window fires, so it is counted. The event at time 4 arrives after window [0,10) fired, so it is late and dropped. Raising LATENESS to 20 would count it, but windows would fire much later.

Sliding windows

With a sliding window, one event counts in several windows. The code below computes "events in the last 30 seconds, every 10 seconds" for a finished list of timestamps.

Python Sliding windows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the output, the sliding windows overlap, so the total of all counts is bigger than the number of events, while the tumbling counts add up to exactly 10.

Lambda and Kappa architectures

Early big-data systems had a fast stream engine that could be wrong and a slow batch engine that was right. The Lambda architecture combines them:

  • a batch layer recomputes accurate results from all the data (slow, high latency),
  • a speed layer computes approximate recent results from the stream,
  • a serving layer merges both for queries.

It works but forces you to write and maintain the same logic twice, once for each engine, and keep the two in agreement.

The Kappa architecture simplifies it: keep everything in a durable log such as Kafka and use one stream-processing code base. To recompute history after a bug fix, replay the log through a new version of the job. Modern engines that can do exact event-time processing, plus cheap long-retention logs and object storage, made Kappa-style designs popular. Batch still wins for heavy historical backfills, large joins and cheap nightly work, so many companies keep both, but share the code where they can (the idea behind Apache Beam and Spark's unified API, see Spark, Flink and What Came After MapReduce).

flowchart LR
    S["Event log"] --> B["Batch layer<br/>(recompute all)"]
    S --> F["Speed layer<br/>(recent only)"]
    B --> V["Serving layer"]
    F --> V
    V --> Q["Queries"]

Lambda: two code paths whose results are merged at query time.

Choosing

  • Need a daily report and hours of delay are fine? Batch.
  • Fraud checks, alerts, live dashboards, recommendations that react in seconds? Stream.
  • Need both a quick estimate and exact historical numbers? Stream for recent data, batch (or replay) for corrections, ideally from one set of business logic.
  • Always decide up front what to do with late data, and test it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should most analytics be grouped by event time rather than processing time?

    Event time reflects when things really happened. Processing time depends on delays in the network and the pipeline.
  2. 2. What does a watermark tell a streaming engine?

    The watermark is a moving estimate of event-time progress. Events older than it that arrive later are considered late.
  3. 3. How many windows does one event belong to in a tumbling window?

    Tumbling windows do not overlap, so each event falls into exactly one. Sliding windows overlap, so an event can fall into several.
  4. 4. What is the main drawback of the Lambda architecture?

    The batch layer and the speed layer are separate code bases. Kappa avoids this by replaying the log through one stream job.

Summary

  • Batch processes bounded data for high throughput; streaming processes unbounded data for low latency.
  • Event time says when something happened; processing time says when you saw it. Aggregate by event time.
  • Windows (tumbling, sliding, session) cut a stream into finite pieces; watermarks decide when a window fires; late events are dropped, redirected or used to update results.
  • Lambda keeps separate batch and speed layers; Kappa uses one stream job and replays the log for reprocessing.

Next: Spark, Flink and What Came After MapReduce.