Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 37 ยท Streaming Concepts

Parallelism, keyBy and Key Groups

See how Flink spreads work with parallelism, routes records by key with keyBy, and uses key groups and max parallelism so keyed state can be rescaled safely.

A single machine cannot handle a stream of a million events per second, so Flink splits every operator into parallel subtasks. The hard part is doing that without breaking the meaning of the computation: if you count clicks per user, all clicks of one user must meet in the same counter. This lesson explains parallelism, how keyBy guarantees that meeting, and the key-group trick that lets state be moved when you scale.

It follows Stateless and Stateful Processing.

Four places to set parallelism

Parallelism is the number of parallel subtasks of an operator. In order of increasing priority:

  1. Cluster default: parallelism.default in the Flink configuration.
  2. Command line: flink run -p 8 job.jar.
  3. Environment: env.setParallelism(8), for every operator in the job.
  4. Operator: stream.map(...).setParallelism(2), only that operator.
Java
env.setParallelism(4);
DataStream<String> parsed = lines.map(Parser::parse).setParallelism(4);
parsed.keyBy(Event::userId).sum("amount").setParallelism(8).print().setParallelism(1);

An operator's parallelism cannot exceed the maximum parallelism (see below), and the job needs enough task slots (see Flink Architecture).

What keyBy does

keyBy(selector) partitions a stream logically by the extracted key. Records with equal keys are guaranteed to go to the same subtask, so that subtask can hold a private state per key.

Java
KeyedStream<Click, String> byUser = clicks.keyBy(Click::userId);

Rules for key types: the key must have a stable hashCode() that is the same on every JVM and every run, so arrays and enums (enum hash codes change between JVMs) and objects that rely on Object.hashCode() are unsuitable. Use strings, numbers, records or tuples of those. POJOs used as keys must override hashCode and equals.

Why not just hash modulo parallelism?

The naive routing would be subtask = hash(key) % parallelism. It works until you rescale. If you stop a job at parallelism 3 and restart it at parallelism 4, a key that was on subtask 1 may now belong to subtask 2, so its saved state is on the wrong machine.

Flink avoids this with key groups:

  1. The key space is split into a fixed number of key groups, equal to the job's max parallelism (default: derived from the first parallelism, rounded up to a power of two, at least 128 and at most 32768).
  2. keyGroup = hash(key) % maxParallelism never changes during the life of the state.
  3. Each subtask owns a contiguous range of key groups. Changing parallelism only changes the ranges, so state moves as whole key groups instead of key by key.
flowchart LR
    K["key"] --> H["hash(key) % maxParallelism<br/>= key group"]
    H --> R["range of key groups<br/>owned by a subtask"]
    R --> S["subtask 0..p-1"]

A key always belongs to the same key group; only the group-to-subtask assignment changes when you rescale.

Set it explicitly for jobs you plan to scale later:

Java
env.setMaxParallelism(720);   // many divisors: 1,2,3,4,5,6,8,9,10,12,... all split evenly

Pick a max parallelism that is a comfortable multiple of the parallelism values you might use. It cannot be changed for a job whose savepoint you want to restore, because that would change every key's group. Larger values cost a little extra state metadata.

Stand-in: key groups and rescaling

Java Stand-in: key groups survive a change of parallelism

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

With key groups, state is shipped as whole groups, and a given group's data lives in one subtask at a time. Restoring a savepoint taken at parallelism 2 into a job with parallelism 4 just means each new subtask loads the key groups of its new range.

Other ways to route a stream

keyBy is not the only partitioner. These repartition a non-keyed stream:

Method Routing Use it for
rebalance() Round-robin to all downstream subtasks Fixing skew from the source
rescale() Round-robin within a local group of subtasks Cheaper than rebalance, less network
shuffle() Random Randomised distribution
broadcast() Every record to every subtask Rules or reference data
global() Everything to subtask 0 Rare; a bottleneck by design
forward() One-to-one, same index Default when parallelism is equal
partitionCustom(...) Your own function Special routing

Stand-in: what rebalance does for skew

Java Stand-in: hash routing versus round-robin on skewed data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

rebalance spreads load evenly but cannot be used when records of the same key must stay together. Use it only for stateless work before a keyed step.

Common mistakes

  • Using a field with a changing value as the key.
  • Keying by an enum or array (unstable hashCode); Flink rejects arrays and warns about enums for good reason.
  • Choosing a tiny max parallelism, then being unable to scale beyond it.
  • Using global() or a constant key, which forces all data through one subtask.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which has the highest priority when determining an operator's parallelism?

    An operator-level setting overrides the environment, command-line and cluster defaults.
  2. 2. What does keyBy guarantee?

    keyBy routes by key so a key's state lives in one place.
  3. 3. Why does Flink use key groups instead of hash(key) % parallelism?

    The key group of a key never changes, so rescaling moves complete groups.
  4. 4. What is the number of key groups equal to?

    There are exactly maxParallelism key groups, so it also limits how far a job can scale out.

Summary

  • Parallelism can be set per cluster, command line, environment and operator; the most specific wins.
  • keyBy sends all records of a key to one subtask using stable hashes.
  • Key groups (count = max parallelism) are the unit of state redistribution; set max parallelism with future scaling in mind.
  • rebalance, rescale, shuffle, broadcast redistribute non-keyed streams without key guarantees.

Next: Delivery Guarantees.