Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 39 ยท Delivery Guarantees

Transactions and Exactly-Once

Understand Kafka transactions: transactional.id, read_committed, consume-transform-produce, zombie fencing and where exactly-once stops, with runnable stand-ins.

Idempotent producers stop duplicates caused by retries within one partition (see Batching, Compression and Idempotence). But many applications do something bigger: read a record, compute a result, write the result to another topic and remember how far they got. If the process crashes between "write the result" and "commit the offset", the work is repeated and the output is duplicated. Kafka transactions close that gap by making several writes, across partitions and topics, succeed or fail together.

This lesson covers the producer API, what consumers see, how "zombie" instances are fenced out, and, importantly, what exactly-once does not cover.

Real versus stand-in. The Java code is real usage, shown read-only because the site cannot reach a Kafka cluster. The TryIt blocks are plain JavaScript stand-ins that reproduce the rules (markers, visibility, epochs). They are not the real protocol, which involves a transaction coordinator broker and the internal __transaction_state topic.

A transaction in four steps

Begin Producer starts a transaction.
Write Records go to many partitions.
Offsets Input offsets join the transaction.
Commit or abort One marker per partition decides.

All writes become visible together, or none do.

1 / 2

The producer API

A transactional producer needs a stable transactional.id. Setting it turns on idempotence automatically.

Java
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("transactional.id", "enricher-1");   // unique per logical producer
props.put("key.serializer", StringSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();          // registers the id, fences older instances

try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("orders-enriched", "o-1", "{...}"));
    producer.send(new ProducerRecord<>("audit-log", "o-1", "{...}"));
    producer.commitTransaction();     // both records become visible together
} catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
    producer.close();                 // fatal: this instance cannot continue
} catch (KafkaException e) {
    producer.abortTransaction();      // recoverable: discard and retry
}

initTransactions() is called once per producer. After that, each unit of work sits between beginTransaction() and commitTransaction() (or abortTransaction()). A transaction that stays open longer than transaction.timeout.ms (default 60 seconds) is aborted by the coordinator, so keep transactions short.

Consume, transform, produce

The classic exactly-once loop reads from an input topic and writes to an output topic. The trick is that the input offsets are committed inside the transaction, using sendOffsetsToTransaction, rather than by the consumer:

Java
consumer.subscribe(List.of("orders"));   // enable.auto.commit=false, isolation.level=read_committed

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    if (records.isEmpty()) continue;

    producer.beginTransaction();
    Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
    for (ConsumerRecord<String, String> r : records) {
        producer.send(new ProducerRecord<>("orders-enriched", r.key(), enrich(r.value())));
        offsets.put(new TopicPartition(r.topic(), r.partition()),
                    new OffsetAndMetadata(r.offset() + 1));
    }
    producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
    producer.commitTransaction();
}

Either the output records and the new consumer offsets are committed, or neither is. After a crash the next instance resumes from the last committed offset and finds no half-finished output. Kafka Streams does all of this for you when you set processing.guarantee=exactly_once_v2 (see Kafka Streams and Stream Processing).

A crash with and without a transaction

The stand-in below processes three input records, "crashes" after producing output on the first attempt, then restarts. Without a transaction the output was already visible, so the retry duplicates it. With a transaction the staged output is discarded.

JS Crash before commit

Press Run (or Ctrl + Enter) to see the console output here.

No output.

What consumers see: read_committed

Records of an open or aborted transaction are physically in the partition log. They are stored like any record, followed by a control record (a commit or abort marker) when the transaction ends. Control records take up an offset but are never returned to your code.

Consumers choose their view with isolation.level:

  • read_uncommitted (the default) returns everything, including records of open and aborted transactions.
  • read_committed returns only committed records. It also stops at the last stable offset (LSO), the offset of the first record that belongs to a still-open transaction. Later, already-committed records wait behind it so that order is preserved.

That last point has a practical cost: one long-running transaction on a partition delays everything behind it for read_committed readers.

JS LSO and read_committed

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Zombies and fencing

Imagine instance A of a service hangs in a long garbage-collection pause. The cluster assumes it is dead, a replacement instance B starts and the work moves. Then A wakes up and tries to continue. A is a zombie, and if it could write, you would get duplicates or conflicting results.

Kafka prevents this with the transactional.id and an epoch. Every initTransactions() for the same id bumps the epoch, and the coordinator rejects writes carrying an older epoch with ProducerFencedException. Because the id is stable, the replacement instance must use the same transactional.id as the one it replaces. Since Kafka 2.5, sendOffsetsToTransaction with the consumer's group metadata also fences zombies by consumer-group generation, so you no longer need one transactional id per input partition.

JS Epoch fencing

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The limits of exactly-once

Kafka's guarantee covers Kafka to Kafka: reads from topics, writes to topics and the offsets, all inside Kafka's own transactions. It cannot make the outside world transactional.

  • An email, an HTTP call or a payment inside your processing step happens once per attempt, even if the Kafka side later aborts.
  • A write to an external database is not part of the Kafka transaction. If the database commit succeeds and the Kafka commit fails, you are back to at-least-once.
  • read_uncommitted consumers and consumers of non-transactional producers see no benefit.

For external systems, combine at-least-once delivery with an idempotent handler or a deduplication key, as shown in At-Most-Once, At-Least-Once, Exactly-Once. Transactions also cost latency (extra requests to the coordinator and marker writes) and commit intervals add end-to-end delay, so use them where duplicates are truly harmful, not everywhere.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the consume-transform-produce loop call sendOffsetsToTransaction?

    Committing the input offsets inside the transaction means a crash leaves either both the output and the offsets, or neither, so the retry cannot duplicate output.
  2. 2. A partition has an open transaction at offset 10 and later committed records at offsets 11 to 20. What does a read_committed consumer see?

    read_committed stops at the last stable offset, the first record of an open transaction. Later committed records wait behind it to preserve order.
  3. 3. How does Kafka stop a zombie producer?

    initTransactions bumps the epoch for the id. A producer with an older epoch gets ProducerFencedException.
  4. 4. Your processing step sends an email and then commits the Kafka transaction. What does exactly-once guarantee about the email?

    Kafka transactions cover Kafka reads and writes only. An attempt that aborts after the email has gone out will repeat it on retry.

Summary

  • Transactions make writes to several partitions, plus consumer offsets, commit or abort together.
  • A transactional producer needs a stable transactional.id, calls initTransactions() once and wraps each unit of work in begin and commit.
  • Consumers must use isolation.level=read_committed to hide open and aborted data; they read up to the last stable offset.
  • Epochs fence zombie instances so a replaced producer cannot write.
  • Exactly-once covers Kafka to Kafka only. External effects still need idempotent handlers.

Next, see how Kafka stores all this on disk in Retention, Segments and Disk Layout.