Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 39 ยท Consumers

Consumer Groups and Rebalancing

Understand Kafka consumer groups: partition assignment, range vs round-robin vs cooperative-sticky, heartbeats, session timeout, static membership and scaling limits.

One consumer reading a busy topic eventually falls behind. Kafka's answer is the consumer group: several consumer instances share the same group.id, and the topic's partitions are divided among them. Each partition is read by exactly one member of the group at a time, so the members work in parallel and no record is processed twice by the group. A different group reading the same topic gets its own full copy of the stream, independent of the first.

Groups also give you fault tolerance. If a member dies, its partitions move to the survivors; if you add a member, it takes over a share. The moving of partitions between members is called a rebalance, and understanding it explains most surprising consumer behaviour in production.

Real versus stand-in. The settings and protocol behaviour described here are real Kafka, with config shown read-only. The TryIt blocks are stand-ins written in plain JavaScript: they implement simplified versions of the assignment strategies and a toy group so you can watch partitions move. Real assignors handle more cases (multiple topics with different subscriptions, rack awareness, weights).

One group, many consumers

Partitions are the unit Each goes to exactly one member.
Scale out Add members to share the load.
Fail over A dead member's partitions move.
Independent groups Each group sees every record.
1 / 2

The group coordinator

Every group has a group coordinator: one broker chosen by hashing the group.id onto a partition of the internal __consumer_offsets topic and taking that partition's leader. All members of the group talk to it. The coordinator tracks who is in the group, receives heartbeats and stores the group's committed offsets (see Offsets and Commits).

A rebalance runs like this:

  1. A member joins, leaves or is declared dead. The coordinator starts a new generation of the group.
  2. Members (re)send a join request. One of them is elected group leader and receives the member list.
  3. The leader runs the configured assignor and returns the plan, which the coordinator distributes in the sync step.
  4. Each member starts reading its new partitions, resuming from the group's committed offsets.

Newer Kafka versions also offer a redesigned group protocol in which the broker computes assignments and rebalances become incremental by design. The concepts below still apply: groups, generations, assignments and committed offsets.

Assignment strategies

The setting partition.assignment.strategy lists the assignors a consumer supports. The built-in ones differ in fairness and in how much they move:

  • Range (RangeAssignor): for each topic separately, sort the partitions and the members and hand out contiguous chunks. It keeps the same partition numbers of different topics on the same member, which helps when joining two topics (co-partitioning), but with several topics it can leave the first members with more.
  • RoundRobin: sort all partitions of all subscribed topics and deal them out one by one. This is usually more even.
  • Sticky: like round-robin in balance, but when membership changes it keeps as many existing assignments as possible.
  • CooperativeSticky: sticky plus incremental cooperative rebalancing, explained below.

The code below shows range and round-robin on two topics of three partitions each with two members.

JS Range versus round-robin

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Range gives consumer-1 four partitions and consumer-2 only two, because each topic's odd partition goes to the first member. Round-robin splits them three and three. The last section shows the hard scaling limit: a group can never have more active consumers than partitions. Extra members sit idle as hot standbys. If you need more parallelism, you need more partitions (see Topics and Partitions).

Eager versus cooperative rebalancing

The classic eager protocol is stop-the-world. When a rebalance starts, every member gives up all its partitions, waits for the new plan, and starts again. Even partitions that end up with the same owner stop for a moment, and short processing pauses accumulate across a large group.

Cooperative rebalancing (used by CooperativeStickyAssignor) is incremental. In the first round, members keep their partitions and only the ones that must move are revoked. In a second round those partitions are given to their new owners. Most of the group keeps working through the whole event.

The simulation below runs a six-partition topic through a series of events and counts how many partitions each style has to stop.

JS How many partitions stop during a rebalance?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Each time a member joins, the eager style pauses every partition that was being read, while the cooperative style revokes only the ones handed to the newcomer. When a member leaves, nothing needs revoking from the survivors in the cooperative case, because the departed member's partitions were already free.

Heartbeats and timeouts

A consumer proves it is alive in two ways, both configurable:

  • heartbeat.interval.ms (default 3 seconds): a background thread sends heartbeats to the coordinator. If a rebalance is under way, the heartbeat response tells the member to rejoin.
  • session.timeout.ms (default 45 seconds in recent versions): if the coordinator hears no heartbeat for this long, it declares the member dead and rebalances. Keep the heartbeat interval at roughly a third of the session timeout or less.
  • max.poll.interval.ms (default 5 minutes): even with heartbeats flowing, a member that does not call poll in time is considered stuck (see Reading with a Consumer).

A short session timeout detects crashes quickly but triggers false alarms during pauses such as long garbage collections or network blips. A long one is calm but leaves a dead member's partitions unread for longer.

JS How fast is a crash noticed?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Static membership

By default a restarting member looks like a brand new one: it leaves, rejoins and causes two rebalances. If you set a unique group.instance.id per consumer (for example the Kubernetes pod name), the member becomes a static member. When it disappears, nothing happens until session.timeout.ms passes; if it comes back with the same id in time, it simply gets its old partitions back with no rebalance. Rolling restarts of a large group become much calmer. The trade-off is that a genuinely dead static member keeps its partitions blocked for the whole session timeout.

PROPERTIES
group.id=invoice-service
group.instance.id=invoice-service-pod-3
session.timeout.ms=60000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

Handling rebalances in code

When partitions are taken away you often want to commit the offsets of finished work first. The Java client lets you pass a ConsumerRebalanceListener to subscribe:

Java
consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
    public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
        consumer.commitSync(currentOffsets);   // save progress before losing them
    }
    public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
        // open per-partition state, restore caches, and so on
    }
});

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A topic has 6 partitions and one consumer group has 9 members. How many members are active?

    A partition goes to only one member of the group, so at most 6 members can be active. The other three are idle standbys.
  2. 2. What is the main benefit of cooperative rebalancing?

    Eager rebalancing revokes everything; the cooperative protocol revokes only the partitions that change owner, so most of the group never pauses.
  3. 3. What does setting group.instance.id enable?

    A static member that returns with the same id before session.timeout.ms expires gets its old partitions back without a rebalance.
  4. 4. Which assignor keeps existing assignments stable while still balancing, and moves partitions incrementally?

    CooperativeSticky combines sticky assignment (few moves) with incremental cooperative rebalancing (no stop-the-world).
  5. 5. Where does a consumer group's coordinator live?

    The group id is hashed to a partition of __consumer_offsets and that partition's leader broker is the group's coordinator.

Summary

  • A consumer group splits a topic's partitions among its members; each partition has one reader per group, and different groups are independent.
  • A group coordinator broker tracks membership and heartbeats; a group leader computes assignments with the configured assignor.
  • Range can be uneven across topics, round-robin is more even, and sticky/cooperative-sticky minimise movement.
  • Active consumers cannot exceed the partition count; more partitions are the only way to scale a group past that.
  • Heartbeats, session.timeout.ms and max.poll.interval.ms define who is considered dead; static membership avoids rebalances on quick restarts.

Next, learn how the group remembers its progress in Offsets and Commits.