Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 39 ยท Practice and Review

Practice: Exercises with Solutions

Eight graded Kafka exercises on lag, partitioning, consumer groups, commit order, compaction, replication and design, each with a full solution and runnable code.

Reading about Kafka is one thing, reasoning with it is another. This lesson gives you eight exercises that go from simple arithmetic to small design problems. Try each one on paper first, then compare with the solution. Where the idea can be simulated, the solution is a runnable stand-in you can edit and run again.

The exercises are ordered roughly from easy to hard. They cover offsets and lag, partitioning, consumer groups, delivery guarantees, compaction, replication settings and topic design.

Real versus stand-in. The settings, formulas and reasoning in the solutions describe real Kafka behavior. The code blocks are hand-written JavaScript stand-ins (small functions with a toy hash and a simplified assignment), so they show the logic, not the real client or broker.

The eight exercises

Lag Offsets and arithmetic.
Partitioning Keys and adding partitions.
Groups Assignment and idle consumers.
Delivery Commit order and crashes.
1 / 2

Exercise 1: compute the lag (easy)

A topic has three partitions. The log end offsets are [120, 98, 150]. The group billing has committed [100, 98, 90]. What is the lag of each partition, and the total lag? Which partition worries you most?

Solution

Lag per partition is the log end offset minus the committed offset: 120 - 100 = 20, 98 - 98 = 0, 150 - 90 = 60. The total is 80 records. Partition 2 has the largest lag and is the one to investigate: perhaps its consumer is slow, stuck on one expensive record, or the partition is skewed by a hot key. Partition 1 is fully caught up. See Monitoring, Lag and Troubleshooting.

Exercise 2: partitioning and adding partitions (easy to medium)

Use this partitioner: add up the character codes of the key and take the remainder after dividing by the partition count. For the keys ana, ben, eli, mia, kim, ivan, zoe and leo, compute the partition with 3 partitions and with 4 partitions. How many keys change partition when you add one? What does this mean for per-key ordering?

Solution

Run the stand-in. It computes both mappings and counts the moved keys.

JS Solution 2: keys that move when partitions are added

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Seven of the eight keys move. Real Kafka uses murmur2 instead of this toy hash, but the effect is the same: most keys map to a different partition once the count changes. Records already written stay where they are, so new events for a key go to a new partition while its old events sit in the old one. Consumers can then see the key's events out of order across the change. That is why you size partitions for future growth up front, and why adding partitions to a keyed topic needs care (see Topics and Partitions and Message Keys and Partitioners).

Exercise 3: group assignment (medium)

Give each consumer a contiguous block of partitions (range assignment) with consumers sorted by name, and the first consumers receiving one extra partition if the division is uneven. Work out the assignment for 7 partitions and 3 consumers, 6 partitions and 4 consumers, and 3 partitions and 5 consumers. How many consumers sit idle in the last case, and what does it tell you about scaling?

Solution
JS Solution 3: range assignment

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With 7 partitions and 3 consumers the split is 3, 2, 2. With 6 partitions and 4 consumers it is 2, 2, 1, 1. With 3 partitions and 5 consumers, two consumers get nothing. A partition is read by at most one member of a group, so the partition count is the ceiling for parallelism. To scale further you need more partitions, not more consumers (see Consumer Groups and Rebalancing). Real Kafka offers other assignors (round-robin, sticky, cooperative-sticky); the range assignor is only one of them.

Exercise 4: where do loss and duplicates come from? (medium)

A consumer handles records 1 to 6. While handling record 4 the process dies exactly between its two steps: "do the work" and "commit the offset". Predict what a replacement consumer sees if the order is commit-then-work, and if it is work-then-commit. Which record is affected in each case?

Solution
JS Solution 4: commit order and a crash

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Commit first, then work: the offset says record 4 is done, but the work never happened, so record 4 is lost. That is at-most-once. Work first, then commit: the work happened but the offset was not saved, so the replacement sees record 4 again. That is at-least-once, and making the handler idempotent turns the duplicate into a no-op. Exactly-once between Kafka topics needs transactions; side effects in other systems need idempotency (see At-Most-Once, At-Least-Once, Exactly-Once and Transactions and Exactly-Once).

Exercise 5: what survives compaction? (medium)

A compacted topic receives these records in order, as key = value (a null value is a tombstone): a=1, b=1, a=2, c=1, b=null, a=3, d=1, c=null, d=2. After the cleaner has processed all of them, which records remain, and what does the table view look like?

Solution
JS Solution 5: compaction by hand

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Compaction keeps the newest record for each key, so you keep b=null (offset 4), a=3 (5), c=null (7) and d=2 (8). The tombstones stay for a while (delete.retention.ms) so that readers can learn about the deletes, then they too are removed. The table is a = 3, d = 2. A real cleaner works on closed segments in the background and never touches the active segment, so a snapshot of a live topic usually contains more than this minimum (see Log Compaction and Tombstones).

Exercise 6: replication settings (medium to hard)

A topic has replication factor 3 and min.insync.replicas=2. Producers use acks=all.

  1. How many brokers can be down while producers can still write?
  2. What happens to producers when two replicas of a partition are down?
  3. Why is min.insync.replicas=3 with a replication factor of 3 a poor choice?
  4. With acks=1, how could an acknowledged record still be lost?
Solution
  1. One. A write succeeds when the leader plus at least one more in-sync replica have it, which is two replicas. With three replicas, one may be missing.
  2. Only the leader remains in sync, which is below the minimum, so the broker rejects writes with a "not enough replicas" error and producers retry or fail. Reads of already committed data still work. The topic favors consistency over availability.
  3. Then every write needs all three replicas in sync, so a single broker failure blocks writes for that partition, and you lose the fault tolerance replication was meant to give.
  4. The leader acknowledges after writing locally. If it fails before a follower copies the record, a follower that becomes the new leader does not have it, and the record is gone. acks=all together with min.insync.replicas closes that gap (see Replication, Leaders and ISR).

Exercise 7: design a topic (hard)

A ride-sharing app emits a location update from every active driver about once every five seconds. Consumers must (a) draw each driver's path in order and (b) show the newest position of every driver on a map. There are about 20,000 drivers online at peak. Decide the record key, whether the topic should be compacted, and how to think about the partition count.

Solution

Key by the driver id. That keeps each driver's updates in one partition and in order, which is requirement (a). It also spreads load evenly, because there are thousands of keys and no single hot one.

The map in (b) only needs the newest position per driver, which is exactly what a compacted topic provides. But compaction deletes the path history that (a) needs. So the usual answer is two topics: a normal retention-based topic for the full stream (kept for hours or days), and a compacted topic, filled by a small stream-processing job or by the same producers, holding the latest position per driver. Mention retention too: a driver who goes offline should get a tombstone so the map forgets them.

For partitions, estimate throughput: 20,000 drivers every five seconds is about 4,000 records per second, small for Kafka. Choose a count that covers the number of parallel consumers you may want (for example 12 or 24) with room to grow, because adding partitions later reshuffles keys. See Topics and Partitions and Log Compaction and Tombstones.

Exercise 8: choose the guarantee (hard)

For each system, name the delivery guarantee you would aim for and the mechanism: (a) a metrics pipeline counting page views for a rough dashboard, (b) a service that charges credit cards for each order event, (c) a job that reads events and copies them into another Kafka topic with no side effects outside Kafka.

Solution

(a) At-most-once or at-least-once is fine, because small errors do not matter. Simple, fast settings (such as acks=1 or auto-commit) may be acceptable, and an occasional duplicate or loss disappears in the noise.

(b) At-least-once delivery with an idempotent handler: commit after the charge, and give the payment provider an idempotency key such as the order id so a redelivered event cannot charge twice. Kafka cannot make an external side effect exactly-once on its own.

(c) Exactly-once is achievable here with transactions: a transactional producer writes the output records and the consumed offsets in one transaction, and downstream readers use isolation.level=read_committed (see Transactions and Exactly-Once). Kafka Streams enables this with one setting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. The log end offset of a partition is 500 and the group's committed offset is 470. What is the lag?

    Lag is the log end offset minus the committed offset: 500 - 470 = 30 records.
  2. 2. A group has 4 consumers reading a topic with 3 partitions. What happens?

    A partition is assigned to at most one consumer of a group, so one member has nothing to read.
  3. 3. A consumer commits the offset first and then does the work, and dies between the two steps. What is the likely result?

    The saved offset says the record is done, so a replacement consumer starts after it. That is at-most-once behavior.
  4. 4. Why can adding partitions to a keyed topic break per-key ordering?

    The partition is computed from the key and the partition count. When the count changes, many keys move.

Summary

  • Lag is log end offset minus committed offset, per partition; look for the partitions that grow.
  • Changing the partition count moves most keys, so plan partitions ahead for keyed topics.
  • A group can use at most as many consumers as it has partitions.
  • Commit after the work for at-least-once, and make handlers idempotent; commit before the work gives at-most-once.
  • Compaction keeps the newest record per key and tombstones delete keys eventually.
  • acks=all with a sensible min.insync.replicas protects acknowledged writes from a single failure.

Next, rehearse explaining it all in Kafka Interview Questions.