Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Operations and Patterns

Monitoring and Backpressure

Monitor Flink jobs with the web UI and metrics, read busy, idle and backpressured time, find the bottleneck operator, and set alerts for checkpoints, lag and restarts.

A running Flink job is a long-lived service. It needs the same care as any production system: you must know whether it is keeping up, how far behind it is, whether checkpoints succeed and where the slowdown is when it is not. This lesson shows the signals Flink gives you and how to read backpressure, the most common performance symptom.

The metric names and UI behaviour below are real; the runnable programs are Java stand-ins that simulate a pipeline with a slow stage so you can see how the numbers arise. It opens chapter 8, after Deployment Modes and Clusters.

What to watch

Signal Why it matters Typical alert
Job status and restarts (numRestarts, uptime) Crash loops restarts in the last 10 minutes above a threshold
Checkpoint success, duration, size, alignment time Recovery safety failed checkpoints, or duration approaching the interval
Source lag (Kafka consumer lag, pendingRecords) Are you falling behind lag growing for several minutes
Backpressure and busy time Where the bottleneck is operator busy above 90 percent for a sustained time
currentEmitEventTimeLag and watermark lag Data freshness lag above your SLA
JVM: heap, GC time, managed memory Memory trouble long GC pauses, heap near full
Records in and out per second Throughput drop to zero while input continues

Flink exposes metrics through metric reporters configured in the cluster: Prometheus (a scrape endpoint on each process), JMX, Datadog, InfluxDB, Graphite and more. A typical setup is the Prometheus reporter plus Grafana dashboards and alert rules.

YAML
metrics.reporters: prom
metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
metrics.reporter.prom.port: 9249

You can also add your own metrics inside functions:

Java
@Override
public void open(OpenContext ctx) {
    badRecords = getRuntimeContext().getMetricGroup().counter("badRecords");
}
// in map(): if (invalid) badRecords.inc();

The web UI

The Flink web UI (default port 8081) shows the job graph with per-operator records, bytes and backpressure and busy colouring, checkpoint history with sizes and durations, exceptions with timestamps, task manager logs and thread dumps, and (when enabled) flame graphs that show where an operator spends CPU time. When something is slow, start there.

Understanding backpressure

Operators are connected by bounded buffers. If a downstream operator cannot keep up, its input buffer fills, the upstream operator cannot hand over more records and blocks, and the pressure moves back all the way to the source, which then reads more slowly. This is backpressure, and it is a feature: the system slows down instead of crashing with an out-of-memory error. But sustained backpressure means the job is not keeping up with its input.

Each task reports three times, in milliseconds per second:

  • busyTimeMsPerSecond: doing useful work.
  • idleTimeMsPerSecond: waiting for input (nothing to do).
  • backPressuredTimeMsPerSecond: blocked because the output buffers are full.

The three add up to about 1000. The bottleneck is the operator that is busy (close to 1000) while the operators before it are backpressured and the operators after it are idle. An operator that is backpressured is a victim, not the cause.

The simulation runs a three-stage pipeline with a slow middle stage and bounded queues, then prints each stage's busy, idle and blocked share of time.

Java Stand-in: finding the bottleneck with bounded queues

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The source spends most of its time blocked, the sink is mostly idle, and the slow map is almost always busy: the textbook bottleneck picture. Optimizing the source or sink would change nothing.

Fixing a bottleneck

Once you know the busy operator, the usual remedies are:

  1. Increase its parallelism (use setParallelism on that operator or in the cluster config) if the work splits evenly. If it does not, you have skew (Parallelism, Skew and Tuning).
  2. Make the work cheaper: avoid per-record object creation, move heavy parsing out of the hot path, use MapState instead of large values, and batch calls to external systems.
  3. Use asynchronous I/O for lookups in databases or services, so each task keeps many requests in flight instead of waiting for each answer.
  4. Check the sink. A slow database or Kafka cluster shows up as backpressure at the very end of the pipeline.
  5. Look at GC and memory, since long pauses look like a slow operator.

Backpressure also affects checkpoints: barriers travel with the data, so under backpressure they arrive late and checkpoints take long or time out. Unaligned checkpoints and buffer debloating help, but fixing the bottleneck is the real cure.

Java Stand-in: async lookups versus blocking lookups

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Alerts that pay for themselves

A short, high-value list: job not in RUNNING state; more than N restarts in 10 minutes; no successful checkpoint for three intervals; checkpoint duration above 80 percent of the interval; source lag rising for 10 minutes; the watermark more than your SLA behind wall-clock time; and heap usage plus GC time per minute. Alert on symptoms users feel (freshness, failures), and keep the per-operator numbers for diagnosis.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which operator is the bottleneck?

    Backpressured operators are victims. The bottleneck is the first operator that is busy while it cannot finish records fast enough.
  2. 2. What is backpressure?

    Bounded network buffers propagate slowness upstream, all the way to the source.
  3. 3. Why do checkpoints slow down under backpressure?

    Barriers flow through the same buffers as records, so a full pipeline delivers them late.
  4. 4. When is async I/O most useful?

    Async I/O keeps many lookups in flight so latency overlaps instead of adding up.

Summary

  • Monitor job state, restarts, checkpoints, source lag, watermark lag, backpressure and JVM health; export metrics through a reporter such as Prometheus.
  • Backpressure propagates slowness upstream via bounded buffers; busy, idle and backpressured time identify the bottleneck.
  • Fix the busy operator: parallelism, cheaper work, async I/O, a faster sink, memory tuning.
  • Backpressure delays checkpoints; fix the cause, and consider unaligned checkpoints.
  • Alert on symptoms: freshness, failures, restarts, failed checkpoints.

Next: Parallelism, Skew and Tuning