Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 37 ยท Time and Windows

Event Time and Processing Time

Understand the notions of time in Flink: event time, processing time and ingestion time, why late and out-of-order events break wall-clock results, and how to assign timestamps.

"How many orders came in per minute?" sounds simple until you ask: which minute? The minute in which the customer clicked Buy, or the minute in which our server happened to process the event? If a phone loses signal in a tunnel and delivers its events five minutes late, the two answers differ. Choosing the right notion of time is the single most important design decision in a streaming job.

This lesson defines Flink's time notions, shows how wrong the wrong choice can be, and explains how timestamps are attached to events. It continues from Connect, Union and Side Outputs.

The three notions of time

flowchart LR
    E["Event happens<br/>EVENT time"] --> I["Reaches Flink source<br/>INGESTION time"]
    I --> P["Operator processes it<br/>PROCESSING time"]

The same event passes three moments. Delays between them are what make streaming hard.

  • Event time. The timestamp inside the event: when it occurred at the producer (a sensor reading time, the click time). It is the same on every run and every replay. Results are deterministic and correct even when events arrive late or out of order, but you need watermarks to know when it is safe to give a result.
  • Processing time. The wall clock of the machine running the operator at the moment it handles the event. Simplest and lowest latency, needs no watermarks, but results depend on speed and load, differ on every run, and are not reproducible after a restart or a replay.
  • Ingestion time. The time the event entered Flink at the source. It is a middle ground that has largely fallen out of use. (In Flink 2.x there is no separate ingestion time setting; use event time with timestamps assigned at the source if you need it.)
Event time Processing time
Clock Timestamp in the data Machine's wall clock
Handles out-of-order or late data Yes, with watermarks No, order of arrival rules
Reproducible after replay Yes No
Latency May wait for stragglers Lowest
Typical use Analytics, billing, correctness Monitoring, timeouts, quick alerts

Since Flink 1.12, event time is the default; you no longer need setStreamTimeCharacteristic. Whether a window or timer uses event or processing time is decided by the window assigner or timer method you call.

A thought experiment: late events

A phone records three clicks at 10:00:10, 10:00:40 and 10:01:05 but uploads them together at 10:03. Flink receives them in one burst.

  • With processing time, all three counted in the 10:03 minute. The 10:00 minute shows fewer clicks than reality and 10:03 shows a spike that never happened.
  • With event time, they are placed into the minutes 10:00 and 10:01 where they belong, and 10:03 is unaffected.
Java Stand-in: counting per minute by event time and by processing time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Event time attributes the three late clicks to the minutes where they happened, so the answer is the same every time you run it, even on a replay. Processing time piles them into 10:03.

An event-time job needs two things: a timestamp per record and a watermark strategy. Both are given by one object, the WatermarkStrategy, which you pass to the source (preferred) or apply with assignTimestampsAndWatermarks:

Java
WatermarkStrategy<Click> strategy = WatermarkStrategy
    .<Click>forBoundedOutOfOrderness(Duration.ofSeconds(10))
    .withTimestampAssigner((click, recordTimestamp) -> click.epochMillis());

DataStream<Click> clicks = env.fromSource(kafkaSource, strategy, "clicks");
  • The timestamp assigner reads the timestamp (milliseconds since epoch) from your data. If you do not give one, a source such as Kafka may supply the record timestamp from the broker, which is usually the producer's send time or the append time.
  • The watermark generator tells Flink how far event time has progressed. That is the subject of the next lesson, Watermarks.
  • WatermarkStrategy.noWatermarks() means you do not use event time at all.

Timestamps are carried inside the stream record next to the value, so operators such as windows read them without needing the field in your class. In a ProcessFunction, ctx.timestamp() returns it.

Processing time in code

Processing-time windows and timers are chosen by class and method name:

Java
clicks.keyBy(Click::user)
      .window(TumblingProcessingTimeWindows.of(Duration.ofMinutes(1)))   // wall-clock minute
      .sum("count");

ctx.timerService().registerProcessingTimeTimer(ctx.timerService().currentProcessingTime() + 30_000);

Use it where "now" is truly what you care about: a timeout, a heartbeat check or a quick dashboard where occasional error is fine.

When to choose which

Ask: "If I replay yesterday's data, should I get yesterday's answer?" If yes, use event time. Billing, analytics, ML features, windows that join two streams, and anything audited all need event time. Choose processing time for operational monitoring ("no data for 5 minutes") and ultra-low-latency approximations.

Java Stand-in: replaying data changes processing-time results but not event-time results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

A fast replay squeezes six minutes of history into one wall-clock minute, so the processing-time answer collapses, while the event-time answer is identical.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which notion of time uses a timestamp stored in the event itself?

    Event time is when the event happened at the source, recorded in the data.
  2. 2. A mobile client uploads yesterday's events in one burst. Which time gives sensible per-minute counts?

    Processing time would put the whole burst into the current minute.
  3. 3. What must an event-time job provide besides timestamps?

    Watermarks let operators decide when windows and timers may fire.
  4. 4. Which task is best suited to processing time?

    A wall-clock timeout is about now, not about the data's own clock.

Summary

  • Event time is when it happened (in the data); processing time is when Flink handles it; ingestion time sits between and is rarely used.
  • Event time gives correct, reproducible results with late and out-of-order data; processing time is simpler and faster but not reproducible.
  • Event time is the default since Flink 1.12. You provide timestamps and a WatermarkStrategy on the source.
  • Pick event time for analytics and correctness, processing time for timeouts and quick monitoring.

Next: Watermarks.