Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 39 ยท Producers

Batching, Compression and Idempotence

Tune Kafka producer throughput with linger.ms, batch.size and compression, then prevent duplicates with enable.idempotence and sequence numbers. Runnable stand-ins included.

Sending one network request per record would waste almost everything: a request has headers, a round trip and a broker write, while a small record is a few hundred bytes. Kafka producers therefore batch. Records bound for the same partition are collected into a batch, optionally compressed as one block, and shipped in a single request. Batching is the main reason Kafka can move hundreds of thousands of records per second from one producer.

The second half of this lesson covers a different risk: retries. If a producer re-sends a record whose first copy actually arrived, you get a duplicate. Idempotence is the producer feature that prevents this.

Real versus stand-in. The property names and their meanings are real Kafka producer settings, shown read-only. The TryIt blocks are small simulations in plain JavaScript: a batching timeline, a toy compressor and a broker that tracks sequence numbers. Real compression codecs and the real protocol are far more detailed.

Why batch?

Fewer requests One round trip carries many records.
Better compression Similar records shrink together.
Efficient writes Brokers append big sequential chunks.

The price is a little waiting time.

1 / 2

How batching works

Inside the producer, send appends the record to a per-partition buffer called the accumulator. A background sender thread drains the batches that are ready. A batch is ready when either of these is true:

  • it has reached batch.size bytes (default 16 KB), or
  • it has been waiting for linger.ms milliseconds since its first record arrived.

linger.ms is the throughput versus latency dial. With 0, a batch is sent as soon as the sender is free, so it only grows while a previous request is still in flight. With 5 or 20, you deliberately add a few milliseconds of delay so that more records join each batch. Historically the default was 0; recent Kafka versions raised it to a few milliseconds, so check the documentation of the version you run.

Two related limits matter. buffer.memory (32 MB by default) is the total memory for unsent records; if the producer produces faster than the brokers accept, the buffer fills and send blocks for up to max.block.ms before throwing. And batch.size is a ceiling per partition, not a target: a big batch.size with linger.ms=0 does not make the producer wait for it.

Java
props.put("linger.ms", "10");           // wait up to 10 ms to fill a batch
props.put("batch.size", "65536");       // up to 64 KB per partition batch
props.put("compression.type", "zstd");  // compress each batch as a whole

A batching timeline

The simulation below feeds 100 records, one per millisecond, 100 bytes each, into a batcher with a 1,000-byte batch.size. It reports how many requests are needed and how long a record waits on average for its batch to leave, for three linger.ms values. The model is simplified (it ignores network time and in-flight requests), but the trade-off it shows is the real one.

JS linger.ms: fewer requests, a little more waiting

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With linger.ms=0 every record leaves alone: 100 requests and no wait. As the linger time grows, batches fill up and the request count falls, but only until the batch hits its 1,000-byte size limit (ten records here). Beyond that point extra lingering buys nothing, because size triggers the send first. The lesson for tuning: raise linger.ms until batches are mostly full, and no further.

Compression

compression.type can be none, gzip, snappy, lz4 or zstd. The producer compresses a whole batch and the broker stores it compressed, so brokers and consumers also benefit from the smaller size. The consumer decompresses it. Compression gives better ratios for bigger batches of similar records, which is why it pairs well with a non-zero linger.ms.

Rough guidance: lz4 and snappy are fast with moderate savings, zstd reaches higher ratios at a modest CPU cost, and gzip is the slowest of the common choices. Always measure with your own data, because JSON with repeated field names compresses very well and already-compressed payloads (images, encrypted bytes) barely compress at all.

The toy below shows why batch size matters. It "compresses" by charging full price for each word the first time it is seen and one byte for every repeat. That is nothing like a real codec, but it captures the key effect: repeated structure inside a batch is cheap.

JS A toy compressor: bigger batches shrink better

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The duplicate problem

Picture a producer sending a batch. The broker writes it, but the acknowledgement is lost on the way back. The producer sees a timeout, cannot know whether the write happened, and retries (as it should for a retriable error; see Writing a Producer). Now the batch exists twice in the log.

A second problem is reordering. If two batches to the same partition are in flight (max.in.flight.requests.per.connection, default 5) and the first fails and is retried, the second may land first. Without extra protection, a retry can swap the order of records.

Idempotence

Setting enable.idempotence=true fixes both. It is the default in Kafka 3.0 and later clients, as long as the other settings are compatible (acks=all, retries above zero and at most 5 in-flight requests per connection).

The mechanism is simple bookkeeping:

  1. The broker gives each idempotent producer a producer id (PID) when it starts.
  2. For every partition, the producer numbers its batches with an increasing sequence number.
  3. The partition leader remembers the last sequence numbers it accepted for each PID.
  4. If a batch arrives with a sequence number it has already stored, the broker does not write it again; it simply acknowledges it. If a batch arrives with a gap (a sequence number that skips ahead), the broker rejects it with OutOfOrderSequenceException, so ordering can never silently break.

Because of step 4, retries are safe and up to five requests can stay in flight without reordering.

The stand-in below implements that bookkeeping. A producer sends three records; the acknowledgement of the second one is lost, so it retries. First against a broker without idempotence, then one with it.

JS Retry after a lost acknowledgement

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Without idempotence the log holds b twice. With it, the retry carries the same sequence number, the broker recognises it and the log stays clean.

What idempotence does not cover

Idempotence protects one producer session writing to one partition at a time. If the application restarts, it receives a new PID and its sequence numbers start over, so a re-sent record from before the crash would not be recognised. It also says nothing about your application sending the same business event twice on purpose or by a logic bug. Guarantees across restarts and across several partitions need transactions (Transactions and Exactly-Once) or deduplication in the consumer (At-Most-Once, At-Least-Once, Exactly-Once).

A starting configuration

PROPERTIES
acks=all
enable.idempotence=true
compression.type=zstd
linger.ms=10
batch.size=65536
delivery.timeout.ms=120000

Treat this as a starting point, not a recipe. Measure throughput, request rate and latency at your real traffic level, and change one setting at a time.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A batch is sent when it is full or when...

    The sender drains a batch when it reaches batch.size bytes or when linger.ms has passed since the batch was started, whichever happens first.
  2. 2. Why does compression usually work better with larger batches?

    The producer compresses a whole batch. Many similar records share field names and patterns, so the codec finds more repetition.
  3. 3. How does the broker recognise a duplicate from an idempotent producer?

    Each batch carries the producer id and a sequence number. A number the leader has already stored is acknowledged but not written again.
  4. 4. A producer process crashes and restarts. Does idempotence prevent duplicates across the restart?

    Idempotence covers a single producer session. Surviving restarts requires transactions with a transactional.id, or deduplication downstream.

Summary

  • Producers batch records per partition; a batch leaves when it reaches batch.size or has waited linger.ms.
  • A little lingering raises throughput and shrinks request counts at the cost of latency; stop raising it once batches are full.
  • compression.type compresses whole batches, and bigger batches of similar records compress better.
  • enable.idempotence=true (default in current clients) uses a producer id and sequence numbers so retries cannot create duplicates or reorder records within a partition.
  • Idempotence is per session and per partition; transactions or consumer-side deduplication handle the bigger picture.

Now switch sides and read the data in Reading with a Consumer.