Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท Operations and Patterns

Parallelism, Skew and Tuning

Understand Flink parallelism and key groups, detect data skew, fix hot keys with two-phase (salted) aggregation, and tune buffers, state and checkpoints for throughput.

Parallelism is how Flink scales: each operator runs as several parallel instances, and each instance handles a share of the data. When shares are equal, doubling parallelism doubles throughput. When they are not, one instance becomes a bottleneck while the rest sit idle, and no amount of extra hardware helps. That imbalance is data skew.

This lesson explains how keys map to parallel instances, how to detect and fix skew, and the main tuning knobs. Flink API calls are real and read-only; the Java programs compute key-group assignment and skew so you can experiment with the numbers. It follows Monitoring and Backpressure.

Setting parallelism

Parallelism can be set at four levels, from most specific to least:

  1. Operator: stream.map(...).setParallelism(8)
  2. Environment: env.setParallelism(4)
  3. Client at submission: flink run -p 4 job.jar
  4. Cluster default: parallelism.default: 1 in the configuration.

Sources are limited by the number of partitions they read (a Kafka source with 6 partitions can use at most 6 busy readers), while sinks are limited by what the target can accept. Within a keyed operator, parallelism also bounds how finely the key space can be spread.

Key groups: how a key picks an instance

State must be redistributed when you change parallelism, which means moving whole chunks of keys, not single keys. Flink therefore divides the key space into a fixed number of key groups, equal to the job's max parallelism (default derived from the first parallelism, between 128 and 32768). The assignment is:

TEXT
keyGroup = hash(key) % maxParallelism
instance = keyGroup * parallelism / maxParallelism

Each instance owns a contiguous range of key groups. When you rescale from a savepoint, whole key groups move between instances. The consequences:

  • Max parallelism is a ceiling you cannot raise later without losing state compatibility (it is baked into savepoints). Set it deliberately with env.setMaxParallelism(720); numbers with many divisors (like 360 or 720) spread evenly over many parallelism values.
  • You can never run with a parallelism greater than the max parallelism.
  • Keys with a very small number of distinct values limit effective parallelism, whatever you configure.

The stand-in computes the assignment and shows how even or uneven a set of keys is.

Java Stand-in: key groups, instances and load per instance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

With thousands of keys the work spreads almost evenly. With three keys, at most three of four instances ever receive data (here exactly one instance gets nothing), whatever parallelism you configure.

Detecting skew

Skew shows up as one subtask doing much more work than its siblings:

  • In the web UI, one subtask of a keyed operator has a much higher records received count or is busy (red) while others are idle.
  • The same checkpoint is slow because that subtask's state is large or its alignment takes long.
  • Backpressure originates in one subtask, not the whole operator.

Common causes are a hot key (one customer, one device, null or a default value that gets 40 percent of the traffic), a bad key choice (country instead of user id) and uneven source partitions.

Fix 1: choose a better key

Re-think the grouping. Counting by (country, device_id) instead of country multiplies the key count. Sometimes you can pre-aggregate before the shuffle (local aggregation) so each upstream task sends one partial per key instead of every record.

Fix 2: two-phase aggregation with salting

When one key is genuinely hot, salt it. Phase one adds a random suffix to the key, spreading the hot key over several instances, and aggregates partial results. Phase two removes the salt and combines the partials. It works for associative aggregates (sum, count, min, max).

Java
events
    .map(e -> Tuple2.of(e.key() + "#" + ThreadLocalRandom.current().nextInt(8), 1L))   // salt 0..7
    .keyBy(t -> t.f0).sum(1)                                                            // phase 1
    .map(t -> Tuple2.of(t.f0.substring(0, t.f0.lastIndexOf('#')), t.f1))                // drop salt
    .keyBy(t -> t.f0).sum(1);                                                           // phase 2

In SQL, the planner can do this for you: set table.optimizer.agg-phase-strategy to TWO_PHASE, and for count-distinct skew enable table.optimizer.distinct-agg.split.enabled (these combine with mini-batch, which must be on).

Java Stand-in: a hot key before and after salting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The first line shows one instance swallowing roughly 900 of 1000 events; after salting the hot key is spread over several instances, and the recombined total is still exactly 900. (The salting program uses a seeded random generator, so the exact distribution is the same on every run.)

General tuning checklist

  • Right-size parallelism: just enough to meet the throughput target with headroom; more subtasks mean more network connections and bigger checkpoint metadata.
  • Reduce serialization cost: use POJOs, records or Avro rather than Kryo-fallback types. Reuse objects only where the API allows it.
  • State: use RocksDB for large state with incremental checkpoints and enough managed memory; keep values small; use TTL; avoid reading list state repeatedly.
  • Network buffers: more buffers help under bursty load; buffer debloating keeps in-flight data small, which improves checkpoint times.
  • Mini-batching in SQL for aggregation-heavy jobs; async I/O for external lookups; object reuse (pipeline.object-reuse: true) only if your functions never modify or hold input objects.
  • Checkpoint interval and min-pause: enough spacing that normal processing is not starved.
  • Measure before and after: change one thing at a time and compare records per second per core.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a key group?

    Key groups let Flink redistribute state when parallelism changes by moving whole groups.
  2. 2. Why should you choose max parallelism carefully?

    Savepoints contain state organized by key groups, so the count is baked in.
  3. 3. Which symptom suggests skew?

    A hot key concentrates load on one subtask.
  4. 4. How does salting help with a hot key aggregate?

    Phase one aggregates per salted key in parallel, phase two combines the small partial results.

Summary

  • Parallelism is set per operator, environment, client or cluster; sources are limited by partitions.
  • hash(key) % maxParallelism picks a key group; instances own ranges of key groups; max parallelism is a long-term decision.
  • Skew (hot keys, few distinct keys, bad keys) leaves some subtasks overloaded; the web UI shows it.
  • Fix with better keys, local aggregation, or two-phase salted aggregation (SQL: TWO_PHASE, distinct split).
  • Tune serialization, state backend, buffers, mini-batching, async I/O and checkpoints; measure each change.

Next: Patterns: Deduplication, Enrichment and CEP