Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 39 ยท Practice and Review

Kafka Interview Questions

Common Apache Kafka interview questions with model answers: partitions, ordering, consumer groups, delivery guarantees, replication, design and troubleshooting.

Kafka interviews tend to follow a pattern. First come the concepts (what is a partition, what does a consumer group do), then guarantees (ordering, delivery, durability), then design questions ("how would you build this?") and finally troubleshooting ("lag is growing, what do you check?"). This lesson collects the questions that come up most, with short model answers. Click a question to open its answer, but try to say the answer out loud first.

A good answer is short, correct, and names the trade-off. Interviewers are rarely looking for a memorized definition; they want to hear that you know why something is the way it is and what it costs.

Real versus stand-in. The answers describe real Kafka behavior and settings, which you can only try on a real cluster. The two TryIt blocks near the end are small hand-written JavaScript stand-ins for ordering and for idempotent produce, so you can see the idea run. They are not the real broker.

How questions are asked

Concepts Topics, partitions, offsets, groups.
Guarantees Ordering, durability, delivery.
Design Keys, partitions, schemas, patterns.
Troubleshooting Lag, rebalances, errors.
1 / 2

Concepts

What is Kafka, and how does it differ from a traditional message queue?

Kafka is a distributed, durable, append-only commit log used for event streaming. In a classic queue, a message is usually removed once a consumer takes it. In Kafka, records stay on the topic for the retention period whatever consumers do, and each consumer group tracks its own position (offset). That allows replay, many independent readers of the same data, and using the log as a source of truth. See What Is Apache Kafka?.

What are topics, partitions and offsets?

A topic is a named stream of records. It is split into partitions, each an ordered, append-only log stored on a broker. Every record in a partition has an offset, a sequence number that never changes. A record is identified by topic, partition and offset. See Topics and Partitions.

Why is Kafka fast, even though it writes to disk?

It writes and reads sequentially, which disks and the operating system's page cache handle very well. It batches records, can compress batches, and uses zero-copy transfer so data goes from the page cache to the network without extra copies in user space. Consumers usually read recent data that is still in the page cache. See The Commit Log Idea.

What ordering does Kafka guarantee?

Order is guaranteed only within a single partition: records are read in the order they were appended. There is no ordering guarantee across partitions of a topic. To keep related events in order, give them the same key so they land in one partition. See Message Keys and Partitioners.

What is a consumer group, and what limits its parallelism?

A consumer group is a set of consumers sharing a group.id that divide a topic's partitions among themselves. Each partition is read by at most one member of the group at a time, so the maximum useful number of consumers equals the number of partitions; extra consumers stay idle. Different groups are independent and each reads the full topic. See Consumer Groups and Rebalancing.

What is the difference between retention and log compaction?

Retention (cleanup.policy=delete) removes old data by age or size, deleting whole segments regardless of keys. Compaction (cleanup.policy=compact) keeps at least the newest record for every key and removes older records of the same key; a record with a null value (a tombstone) eventually deletes the key. Compaction suits "latest state" topics such as table changelogs. See Retention, Segments and Disk Layout and Log Compaction and Tombstones.

Guarantees

Explain leaders, followers and the ISR.

Every partition has one leader that handles reads and writes and followers that copy its log. The in-sync replicas (ISR) are the leader plus the followers that are caught up closely enough. If the leader fails, a new leader is chosen from the ISR, so no acknowledged data is lost as long as acknowledgements were required from the ISR. See Replication, Leaders and ISR.

What do acks and min.insync.replicas do together?

acks=0 does not wait for any acknowledgement, acks=1 waits for the leader only, and acks=all waits for all in-sync replicas. min.insync.replicas is the minimum size of the ISR for acks=all writes to succeed. With replication factor 3, acks=all and min.insync.replicas=2, a write is acknowledged only when at least two replicas have it, and the partition tolerates one broker failure without losing acknowledged data or blocking writes.

What are at-most-once, at-least-once and exactly-once delivery?

At-most-once: a record is never processed twice but may be lost, for example when a consumer commits before processing. At-least-once: nothing is lost but duplicates are possible, for example when it commits after processing and crashes before the commit. Exactly-once: each record's effect happens once; inside Kafka this is achieved with idempotent producers and transactions, while external side effects need idempotent handlers. See At-Most-Once, At-Least-Once, Exactly-Once.

What does an idempotent producer do?

With enable.idempotence=true the producer gets an id and numbers each batch per partition. The broker remembers the latest sequence number and ignores a retry it has already stored, so a retry after a lost acknowledgement does not create a duplicate within that producer session and partition. In current Kafka clients it is on by default together with acks=all. See Batching, Compression and Idempotence.

How do Kafka transactions work, and what are their limits?

A producer with a transactional.id can write to several partitions and commit consumed offsets in one atomic transaction; consumers set isolation.level=read_committed so they only see committed records. A zombie producer with the same id is fenced off. The guarantee covers data in Kafka; a database update or an HTTP call inside the processing step is not part of the transaction. See Transactions and Exactly-Once.

What happens during a consumer group rebalance?

When members join, leave or time out, or the partition count changes, the group coordinator reassigns partitions. With the older eager protocol all consumers revoke everything and pause; the cooperative-sticky assignor moves only the partitions that must change. Frequent rebalances hurt throughput, so keep processing in a poll loop short, tune max.poll.interval.ms, and consider static membership. See Consumer Groups and Rebalancing.

What is the role of ZooKeeper, and what is KRaft?

Older Kafka versions stored cluster metadata in an external ZooKeeper ensemble. KRaft mode replaces it with a Raft-based controller quorum inside Kafka itself, which simplifies operations and scales to more partitions. ZooKeeper mode has been removed from current major versions, so new clusters run KRaft. See Running a Cluster: KRaft, Brokers and Controllers.

Design

How do you choose the number of partitions?

Work backward from the target throughput and the parallelism you need. Measure what one partition can sustain for producing and for one consumer, and divide the target by the smaller number. Add headroom, because adding partitions later changes the key-to-partition mapping and can break per-key order. Too many partitions cost memory, file handles, longer leader elections and more end-to-end latency. See Topics and Partitions.

How do you choose a record key?

Use the identifier of the entity whose events must stay in order, such as a customer, account or device id. It should have many distinct values so the load spreads, and no single value should dominate. A hot key sends too much traffic to one partition. If ordering does not matter, use no key and let the producer spread records.

How would you guarantee no data loss?

On the producer: acks=all, idempotence on, retries enabled and handling of send failures. On the topic: replication factor 3 and min.insync.replicas=2, with unclean leader election off. On the consumer: disable auto-commit or commit only after processing, and make handlers idempotent. Monitor under-replicated partitions and lag.

How do you handle a message that always fails (a poison message)?

Do not let it block the partition forever. Retry a bounded number of times, possibly through retry topics with a delay, and then publish the record with error details to a dead letter topic, commit past it, and alert. Someone can inspect and replay it later. See Retries, Dead Letter Topics and Poison Messages.

How do you evolve a message format safely?

Use a schema format such as Avro or Protobuf with a schema registry and choose a compatibility mode: backward-compatible changes (such as adding an optional field with a default) let new consumers read old data. Decide whether to upgrade consumers or producers first according to the mode. See Schemas, Avro, Protobuf and JSON.

Kafka or a message broker like RabbitMQ?

Pick Kafka for high-throughput streams, replay, several independent consumers of the same data and stream processing. Pick a broker for rich routing, per-message acknowledgement and priorities, and short-lived task queues where replay is not needed. Many systems use both. See What Is Apache Kafka?.

Troubleshooting

Consumer lag keeps growing. What do you check?

First find out whether it is one partition or all. One partition points to a hot key or a stuck consumer; all partitions point to a slow consumer or too little parallelism. Check the processing time per record (a slow database call), the number of consumers against partitions, frequent rebalances, and the fetch settings. Scale consumers up to the partition count, speed up the handler, or add partitions. See Monitoring, Lag and Troubleshooting.

Why do I see duplicates in the consumer?

Likely causes are a producer retry without idempotence, a consumer that processed records but crashed before committing, or a rebalance that handed a partition over before the commit. The remedy is idempotent handlers (a unique key or version check), not hoping duplicates never occur.

The producer fails with "not enough replicas". What does it mean?

The number of in-sync replicas for the partition dropped below min.insync.replicas while the producer used acks=all. Brokers are down or lagging. Fix the brokers or network. Lowering the minimum "fixes" the error but reduces durability.

A large message is rejected. Which settings matter?

The size limit appears in several places: the producer's max.request.size, the broker's message.max.bytes (or the topic's max.message.bytes), the replica fetch size, and the consumer's max.partition.fetch.bytes and fetch.max.bytes. All must allow the size. Often the better fix is to store large payloads in object storage and send a reference.

What does "under-replicated partitions" mean, and why does it matter?

It means some partition has followers missing from the ISR. Durability is reduced, and one more failure may lose availability or data. Causes include a failed broker, a slow disk or network, or heavy load. Treat sustained under-replication as an alert.

Two small demos

Two ideas that interviewers love to probe are ordering and idempotent produce. The stand-ins below show them in a few lines.

The first one sends events from two users through two partitions. Each user keeps a strict order, but a consumer reading both partitions sees an interleaving, so a global order does not exist.

JS Ordering holds per partition only

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The second shows how sequence numbers protect against duplicate writes from retries. The broker remembers the last sequence number per producer and partition: a repeat is ignored, a gap is rejected.

JS Idempotent produce with sequence numbers

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A topic has 6 partitions and a consumer group has 8 consumers. How many consumers actually receive records?

    Each partition goes to one member of the group, so at most 6 consumers are active and 2 are idle.
  2. 2. Which combination gives the strongest durability for acknowledged writes on a replication factor 3 topic?

    With acks=all and at least two replicas required, an acknowledged record exists on two brokers, so one failure cannot lose it.
  3. 3. How do you keep all events for one customer in order?

    Equal keys map to the same partition, and a partition preserves the order of its records.
  4. 4. A consumer crashes after processing a record but before committing its offset. What will the next consumer of that partition do?

    It starts from the last committed offset, so the record is delivered again. That is at-least-once, and the handler should be idempotent.
  5. 5. What is the best first question when consumer lag is growing?

    One partition suggests a hot key or a stuck consumer, while all partitions suggest a slow handler or too little parallelism.

Summary

  • Order is per partition; use the key to group related events.
  • A group reads each partition with at most one member, so partitions cap parallelism.
  • Durability comes from replication plus acks=all plus min.insync.replicas.
  • Delivery semantics depend on where you commit; idempotent handlers make duplicates harmless.
  • Idempotent producers and transactions give exactly-once inside Kafka, not for external systems.
  • For troubleshooting, narrow the problem first: which partition, which group, which broker.

Finish with the quick reference in the Kafka Cheat Sheet.