Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 39 ยท Consumers

Reading with a Consumer

Learn the Kafka consumer poll loop: subscribe vs assign, deserializers, auto.offset.reset, max.poll.records and graceful shutdown, with a runnable poll-loop stand-in.

A consumer reads records from Kafka topics. Unlike a message queue, Kafka does not push records to you and does not delete them once you have read them. Your consumer pulls: it asks a broker for the next records after a position it keeps track of, processes them, and asks again. Many consumers can read the same topic independently, each with its own position.

This lesson covers the basic shape of that loop: how to start it, what each poll returns, where a brand new consumer begins reading, and how to stop cleanly. The next lessons build on it with consumer groups and offset commits.

Real versus stand-in. The consumer code is real Java and JavaScript client usage, shown read-only because the site cannot connect to a Kafka cluster. The TryIt blocks are plain JavaScript stand-ins: a small log, a consumer with a position, and a poll loop that behaves like the real one in the ways this lesson describes.

The consumer loop

Subscribe Pick topics, join a group.
Poll Fetch the next batch of records.
Process Your business logic.
Commit Remember how far you got.

Repeat until shutdown, then close cleanly.

1 / 2

A consumer in Java

Java
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "invoice-service");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("auto.offset.reset", "earliest");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("orders"));

    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> record : records) {
            System.out.printf("%s-%d@%d key=%s value=%s%n",
                record.topic(), record.partition(), record.offset(),
                record.key(), record.value());
        }
    }
}

The pieces:

  • group.id names the consumer group this consumer belongs to. Consumers sharing a group id share the work of a topic; different groups each get every record.
  • Deserializers are the mirror image of the producer's serializers: bytes in, objects out. They must match what the producer wrote. A mismatch (for example, reading JSON bytes with an integer deserializer) throws a SerializationException when the record is decoded.
  • poll(Duration) is the heart of the consumer. It returns whatever records are available, waiting up to the timeout if there are none. It also keeps the consumer alive in the group and sends heartbeats in the background, so you must keep calling it.
  • Each record carries its topic, partition, offset, key, value, timestamp and headers.

A JavaScript client works the same way conceptually, though the library hides the loop. With KafkaJS, for example:

JavaScript
const consumer = kafka.consumer({ groupId: "invoice-service" });
await consumer.connect();
await consumer.subscribe({ topic: "orders", fromBeginning: true });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log(partition, message.offset, message.key?.toString(), message.value?.toString());
  },
});

Option names vary between libraries, so check the one you use. Notice fromBeginning: that is this client's name for the setting you will meet next.

subscribe versus assign

subscribe(topics) joins a consumer group and lets the group coordinator decide which partitions each member reads. If a member joins or leaves, partitions are rebalanced. You can even subscribe with a regex pattern, such as every topic starting with orders-.

assign(partitions) skips groups entirely: you name the exact topic partitions to read, and nothing rebalances. That is useful for tools that must read one specific partition, for replay utilities, or when you manage coordination yourself. You cannot mix the two on one consumer instance.

Where does a new consumer start?

When a consumer group has never committed an offset for a partition, or the committed offset no longer exists (for example, retention deleted it), the consumer needs a starting point. auto.offset.reset decides:

Value Meaning
latest (default) start at the end; only records that arrive afterwards are seen
earliest start at the oldest record still in the log
none throw an exception so your code decides

The setting only applies when there is no valid committed offset. Once the group has committed a position, restarts resume from it and auto.offset.reset is ignored.

The stand-in below keeps a one-partition log. Five records exist before the consumer first arrives and two more come in later. A poll hands out at most maxPollRecords records and moves the consumer's position past them, just like the real client.

JS A poll loop and auto.offset.reset

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice the last line: the restarted consumer uses latest, yet it starts from the committed position and sees only the one record added since. The reset policy is just a fallback.

Controlling a poll

A few settings shape each poll:

  • max.poll.records (default 500) caps the number of records one call returns.
  • fetch.min.bytes and fetch.max.wait.ms let the broker wait to gather more data per fetch, trading latency for fewer requests, the consumer-side twin of linger.ms.
  • max.poll.interval.ms (default 5 minutes) is the longest allowed gap between two poll calls. If your processing of one batch takes longer, the group assumes you are stuck, removes you and rebalances. Lower max.poll.records or speed up processing if you hit it.

Stopping cleanly

A consumer should leave its group deliberately. Calling close() commits (if auto-commit is on), tells the coordinator you are leaving and lets the other members take over your partitions immediately instead of waiting for a timeout. Because a KafkaConsumer is not thread-safe, the only method you may call from another thread is wakeup(), which makes a blocked poll throw a WakeupException:

Java
final Thread mainThread = Thread.currentThread();
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    consumer.wakeup();
    try { mainThread.join(); } catch (InterruptedException ignored) { }
}));

try {
    while (true) {
        for (ConsumerRecord<String, String> r : consumer.poll(Duration.ofMillis(500))) {
            handle(r);
        }
        consumer.commitSync();
    }
} catch (WakeupException expected) {
    // shutting down
} finally {
    consumer.close();   // leave the group cleanly
}

The next stand-in models that pattern with a flag. A "signal" arrives while the loop is running; the loop finishes the batch it holds, commits and closes. Records that were processed are never lost or repeated by the shutdown.

JS Graceful shutdown

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Multi-threaded consumers

KafkaConsumer is not thread-safe. Every method except wakeup() must be called from the one thread that owns the consumer; sharing it gives ConcurrentModificationException. That leaves two patterns when one thread cannot keep up.

One consumer per thread. Each thread creates its own consumer in the same group; Kafka assigns them different partitions. It is simple, keeps ordering per partition and needs no extra commit logic. The limit is the partition count: with 6 partitions, a 7th consumer idles.

Java
ExecutorService pool = Executors.newFixedThreadPool(3);
for (int i = 0; i < 3; i++) {
    pool.submit(() -> {
        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));
            while (running.get()) {
                for (ConsumerRecord<String, String> r : consumer.poll(Duration.ofMillis(500))) {
                    handle(r);
                }
            }
        }
    });
}

One poller, a pool of workers. When processing is slow (a database call, an HTTP request) but you want more parallelism than partitions, one thread polls and hands records to workers. Two rules make it safe. Offsets may only be committed after the worker finishes, otherwise a crash loses records. And the poller must keep calling poll() so it is not kicked out of the group, using pause() to stop fetching while workers are busy and resume() when they catch up. Keep records of one key (or one partition) on one worker if you need ordering.

JS Pause, resume and commit after processing

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A third option is to offload work to a framework that does this for you: Spring's ConcurrentMessageListenerContainer is "one consumer per thread" with a concurrency setting, and parallel-consumer style libraries add per-key worker pools. Whatever you choose, the rule stays: the thread that polls owns the consumer and the commits.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does auto.offset.reset control?

    It is only a fallback for the case of no committed position (or one that is out of range). With a valid committed offset, the consumer simply resumes from it.
  2. 2. What is the difference between subscribe and assign?

    subscribe uses the group coordinator and rebalances when members change. assign gives you manual control and no group membership management.
  3. 3. Which KafkaConsumer method may be called safely from another thread?

    KafkaConsumer is not thread-safe, with wakeup() as the one exception. It makes a blocked poll throw WakeupException so the loop can shut down.
  4. 4. Processing one poll batch takes longer than max.poll.interval.ms. What happens?

    The interval between polls is how the group knows a member is alive and making progress. Exceeding it triggers removal and rebalancing.

Summary

  • A consumer pulls records in a poll loop and keeps its own position; reading never deletes data.
  • group.id plus subscribe shares partitions across a group; assign takes explicit partitions without rebalancing.
  • Deserializers must match the producer's serializers.
  • auto.offset.reset (latest, earliest, none) only matters when no valid committed offset exists.
  • Keep calling poll within max.poll.interval.ms, and shut down with wakeup() and close().

Next, see how several consumers share one topic in Consumer Groups and Rebalancing.