Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 37 ยท Streaming Concepts

Backpressure and Flow Control

Understand backpressure in Flink: how credit-based flow control slows fast producers, how to spot the bottleneck in the web UI, and the usual fixes for slow sinks and skew.

What happens when a source produces 100,000 events per second but an operator downstream can only handle 60,000? Data cannot pile up forever, and dropping it silently would break correctness. Flink's answer is backpressure: the slow part pushes back on the fast part so the whole pipeline runs at the speed of its slowest stage.

Backpressure is not an error. It is the system working as designed. But sustained backpressure tells you that something is a bottleneck, and the main skill is finding which operator that is. This lesson follows Delivery Guarantees.

Between tasks, records travel in fixed-size network buffers. Every task has a limited pool of them.

  1. A sending subtask fills output buffers and hands them to the network.
  2. The receiving subtask announces credit: how many free input buffers it has.
  3. The sender may only send as many buffers as it has credit for.
  4. If the receiver is slow, its buffers stay full, it announces no credit, the sender's output buffers fill and the sender blocks.
  5. That sender's own upstream neighbour then loses credit in turn, and the effect travels all the way up to the source, which simply reads less from Kafka (the data stays safely in the log).
flowchart LR
    S["Source<br/>slows down"] --> M["Map<br/>output buffers full"]
    M --> W["Window<br/>input buffers full"]
    W --> K["Slow sink<br/>the real bottleneck"]

Backpressure travels upstream from the slow operator to the source.

This is called credit-based flow control. Its key property: no unbounded queues, no data loss, and no extra configuration; memory use stays predictable.

Stand-in: a bounded buffer blocks a fast producer

Java Stand-in: producer slowed by a slow consumer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The producer could emit all eight events instantly, but it spends most of its time blocked, so the pipeline finishes at the consumer's pace. That is backpressure without data loss.

Finding the bottleneck

In the web UI, open the job and look at each operator box:

  • Backpressured (blue-red scale): the percentage of time the task was blocked waiting for output buffers. A high value means that downstream is slow.
  • Busy (CPU-like gauge): the time the task spent doing work. A task at 100% busy is a bottleneck candidate.
  • Idle: waiting for input; the sign of a starved task.

The recipe: starting from the source, follow the backpressured operators downstream. The first operator that is busy but not backpressured is the bottleneck. Its downstream neighbours are idle because it starves them.

Also look at per-subtask values: if one of eight subtasks is 100% busy and the rest idle, you have data skew (see The Dataflow Model), not a lack of capacity. Metrics such as backPressuredTimeMsPerSecond, busyTimeMsPerSecond and idleTimeMsPerSecond give the same values for dashboards.

Usual causes and remedies

Cause Symptom Fix
Slow external call in a map or sink (database, HTTP) Busy sink, low CPU Use Async I/O, batch writes, a faster sink
Data skew (hot key) One subtask busy Salt the key, pre-aggregate, rebalance before keyed step
Too little parallelism All subtasks of an operator busy Raise parallelism, check slots
Expensive per-record logic (regex, JSON) High CPU in an operator Optimise, cache, move out of hot path
Slow state access High RocksDB latency, disk I/O Faster disks, tune RocksDB, smaller state, state TTL
Long GC pauses Periodic stalls on one TaskManager More memory, fewer objects, tune GC
Checkpoints with huge state Backpressure during checkpointing Incremental checkpoints, unaligned checkpoints
Slow downstream system Sink backpressured, external system saturated Throttle, scale the target, buffer in Kafka

Remember that adding parallelism only helps if the bottleneck can be split. A single hot key, a single-row sink or global() partitioning stays at one subtask however many slots you add.

Unaligned checkpoints and backpressure

Under heavy backpressure, aligned checkpoints suffer: a barrier is stuck behind full buffers, so the checkpoint takes very long and may time out. Unaligned checkpoints (env.getCheckpointConfig().enableUnalignedCheckpoints()) let the barrier overtake the queued records and snapshot those in-flight records as part of the state. Checkpoints stay fast, at the cost of larger snapshots.

Stand-in: finding the bottleneck from busy times

Java Stand-in: which operator is the bottleneck?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Source and Parse look slow only because they are blocked; the real bottleneck is Enrich, which is busy and not blocked. Fixing the sink would change nothing.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does backpressure mean in Flink?

    Credit-based flow control blocks senders when receivers have no free buffers; nothing is dropped.
  2. 2. How do you find the bottleneck operator in the UI?

    Operators above it show backpressure because they wait for it; it is busy while those below it are idle.
  3. 3. One subtask of an operator is 100% busy and the others are idle. What is the likely cause?

    A hot key sends most records to one subtask. More parallelism will not fix it.
  4. 4. What is a good fix for a sink that waits on a slow external database for every record?

    Batching and async calls hide latency so the operator is not idle while waiting.

Summary

  • Flink uses credit-based flow control: a slow receiver stops granting credit, senders block and the effect travels upstream to the source.
  • Backpressure is normal and lossless; sustained backpressure signals a bottleneck.
  • Find it with the busy and backpressured metrics, looking for the first busy, non-backpressured operator and checking per-subtask skew.
  • Remedies: async I/O and batching, parallelism, key redesign, tuned state access, and unaligned checkpoints for slow checkpoints under load.

Next, start writing code in the DataStream API: DataStream Basics: Environment, Sources and Sinks.