Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 39 ยท Producers

Writing a Producer

Write a Kafka producer in Java and JavaScript: records, serializers, send(), callbacks, acks, retriable errors and delivery timeouts, with a runnable producer stand-in.

A producer is any program that appends records to Kafka topics. It might be a web service that publishes "order placed", a sensor gateway pushing readings, or a job that copies rows out of a database. The client library hides a lot of work: it turns your objects into bytes, picks a partition, groups records into batches, finds the leader broker for each partition and retries when something goes wrong.

This lesson shows what a producer looks like in code, what send really returns, and how errors and retries behave. The next lesson tunes batching and idempotence.

Real versus stand-in. The Java and JavaScript client code below is real usage, shown read-only because the site cannot reach a Kafka cluster. The TryIt blocks are hand-written stand-ins in plain JavaScript that copy the producer's shape: records, a serializer, a partitioner, futures and acks. They are not the real client.

What a producer does

Build a record Topic, key, value, headers.
Serialize Objects become bytes.
Pick a partition By key, or spread evenly.
Batch and send To the partition leader.

send() returns before the broker has answered.

1 / 2

The anatomy of a record

A ProducerRecord has a topic and a value, and optionally a key, a partition, a timestamp and headers. The key decides which partition the record lands in (see Message Keys and Partitioners); headers carry metadata such as a trace id or a content type without touching the value.

The broker never interprets keys or values. To Kafka they are just bytes, so the producer needs a serializer for each: a function from your object to a byte array. Kafka ships serializers for strings, integers, longs, doubles, byte arrays and a few more. For structured data you normally use JSON, Avro or Protobuf (see Schemas, Avro, Protobuf and JSON).

A producer in Java

Java
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");

try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    ProducerRecord<String, String> record =
        new ProducerRecord<>("orders", "customer-42", "{\"id\":1001,\"total\":19.90}");

    producer.send(record, (metadata, exception) -> {
        if (exception != null) {
            System.err.println("send failed: " + exception);
        } else {
            System.out.printf("stored in %s-%d at offset %d%n",
                metadata.topic(), metadata.partition(), metadata.offset());
        }
    });
}   // close() flushes everything that is still buffered

bootstrap.servers is only a starting list. The client contacts any broker in it, asks for the cluster metadata and then talks to whichever brokers lead the partitions it needs. You do not have to list every broker.

send is asynchronous. It puts the record into an in-memory buffer and returns a Future<RecordMetadata> immediately. The metadata (topic, partition, offset, timestamp) only exists once the broker has accepted the record. Closing the producer with close() waits for buffered records to be sent, which is why the try-with-resources block above matters: exit the program without flush() or close() and the last records may never leave your process.

Three ways to send

The same send call supports three styles. Pick the one that matches how much you care about each record.

Java
// 1. Fire and forget: ignore the result. Fastest; failures after the retries are lost silently.
producer.send(record);

// 2. Synchronous: block until the broker answers. Simple, but one record in flight at a time.
try {
    RecordMetadata meta = producer.send(record).get();   // throws if delivery failed
    System.out.println("offset " + meta.offset());
} catch (InterruptedException | ExecutionException e) {
    System.err.println("send failed: " + e.getCause());
}

// 3. Asynchronous with a callback: keep sending, handle each result later.
producer.send(record, (meta, ex) -> { if (ex != null) { /* log, alert, retry elsewhere */ } });

Fire and forget suits metrics where losing a sample does not matter. Synchronous sends are fine for scripts and tests but cap your throughput at one round trip per record. The callback style gives you both speed and error handling, so it is the usual choice in services.

The producer workflow in order

  1. The client is created from its configuration and fetches cluster metadata from bootstrap.servers.
  2. send builds a ProducerRecord and runs any interceptors.
  3. The key and value are serialized to bytes.
  4. The partitioner chooses the partition (hash of the key, or sticky batching when there is no key).
  5. The record joins a per-partition batch in the accumulator.
  6. The sender thread ships full or lingering batches to each partition leader and the broker answers with the offset or an error.
  7. Retries, the callback or the Future complete the story.

Key producer settings

Setting Typical value What it controls
bootstrap.servers broker1:9092,broker2:9092 Where to find the cluster first.
key.serializer, value.serializer String, JSON, Avro... How objects become bytes.
acks all How many replicas must confirm.
compression.type none, gzip, snappy, lz4, zstd Compression of whole batches.
batch.size 16 KB Maximum bytes per partition batch.
linger.ms 0 to 20 How long to wait for a batch to fill.
buffer.memory 32 MB Total memory for unsent records.
enable.idempotence true No duplicates from retries.

A producer in JavaScript

JavaScript has no official client in the Kafka project. Popular libraries include KafkaJS and Confluent's JavaScript client, and their APIs are similar. A KafkaJS-style producer looks like this:

JavaScript
const { Kafka } = require("kafkajs");

const kafka = new Kafka({ clientId: "order-service", brokers: ["localhost:9092"] });
const producer = kafka.producer();

await producer.connect();
const result = await producer.send({
  topic: "orders",
  messages: [{ key: "customer-42", value: JSON.stringify({ id: 1001, total: 19.9 }) }],
});
console.log(result); // partition and offset information per message batch
await producer.disconnect();

Here send returns a promise that resolves when the broker has acknowledged the records, so await gives you the "wait for the result" style. Check your library's documentation for exact option names; the concepts are the same across clients.

Acknowledgements: acks

The acks setting controls when the broker says "done":

acks The leader answers... Risk
0 immediately, without even writing records can vanish silently
1 after writing to its own log lost if the leader dies before followers copy it
all (or -1) after all in-sync replicas have the record safest

Current Kafka clients default to acks=all. How many replicas must be in sync for that to succeed is a topic setting, min.insync.replicas, explained in Replication, Leaders and ISR.

A producer stand-in you can run

The stand-in below keeps a two-partition topic in memory. The "client" serializes the key and value, chooses a partition from a hash of the key (a simple stand-in, not Kafka's real murmur2 hash) and returns a promise of {partition, offset}. It then sends the same kind of record in the three styles: fire and forget, awaited, and with a callback.

JS Three ways to send

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Both customer-1 records end up in the same partition, in the order they were sent. That is the property the real producer gives you for a given key, and it is why keys matter.

Errors and retries

Things go wrong in two different ways.

Retriable errors are temporary: a leader election is in progress (NOT_LEADER_OR_FOLLOWER), a request timed out, or the network blinked. The producer retries these on its own. Its patience is bounded by delivery.timeout.ms (two minutes by default), which covers the whole life of a record from send until success or final failure. retries defaults to a huge number, so in practice the timeout is what stops retrying, and retry.backoff.ms sets the pause between attempts. request.timeout.ms limits one attempt.

Non-retriable errors will fail the same way every time, so the client reports them at once: RecordTooLargeException (the record exceeds max.request.size or the broker's limit), a SerializationException, or an authorization failure. Retrying cannot help; you must fix the record or the configuration.

When delivery finally fails, your callback gets the exception (or the future throws). Decide in advance what your application does then: log and alert, write the record to a local fallback store, or stop accepting work. Ignoring failures is how data is lost quietly.

Inside the producer: accumulator and sender

send() does little on your thread. It runs interceptors, serializes the key and value, picks a partition and appends the record to the record accumulator, an in-memory set of per-partition batches. A separate sender thread drains ready batches, groups them by destination broker, sends the requests and handles responses, retries and callbacks.

The life of a record

Interceptor Optional hook before send.
Serialize and partition On your thread.
Accumulator Batches wait in buffer.memory.
Sender thread Ships batches to leaders.

Your callback runs on the sender thread, so keep it short.

1 / 2

Settings that govern this pipeline:

  • buffer.memory (32 MB): total space for batches waiting to be sent. When it is full, send() blocks.
  • max.block.ms (60 s): how long send() (and partitionsFor()) may block waiting for buffer space or metadata before throwing TimeoutException. A producer that fails after exactly a minute usually hit this: the cluster is unreachable or the buffer is full.
  • delivery.timeout.ms (2 min): the budget from the moment send() returns until success or failure is reported. It should be at least linger.ms + request.timeout.ms.
  • max.request.size (1 MB) limits one request, and so one record. The broker's message.max.bytes (topic-level max.message.bytes) limits one batch. Raising only one side still fails with RecordTooLargeException or MESSAGE_TOO_LARGE. Better than raising limits: store large files elsewhere and send a reference.

Interceptors implement ProducerInterceptor and are listed in interceptor.classes. onSend can read or change a record (add a trace-id header), and onAcknowledgement sees every result. They suit cross-cutting concerns such as tracing and metrics; keep them fast, because onSend runs on your thread and onAcknowledgement on the sender thread.

Java
public class TraceInterceptor implements ProducerInterceptor<String, String> {
    public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
        record.headers().add("trace-id", UUID.randomUUID().toString().getBytes(StandardCharsets.UTF_8));
        return record;
    }
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) { /* count successes and errors */ }
    public void close() {}
    public void configure(Map<String, ?> configs) {}
}
// props.put(ProducerConfig.INTERCEPTOR_CLASSES_CONFIG, TraceInterceptor.class.getName());
JS A full buffer blocks send()

Press Run (or Ctrl + Enter) to see the console output here.

No output.

For the three send styles (fire-and-forget, synchronous, asynchronous), see the stand-in earlier in this lesson. For tuning batches, see Batching, Compression and Idempotence.

A retry stand-in

This stand-in models a flaky broker with attempts as the only clock. The first two attempts fail with a retriable error, and the third succeeds. A second record is too large, which is non-retriable and fails immediately. A third runs out of its delivery budget.

JS Retriable and non-retriable errors

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A real producer measures its budget in milliseconds rather than attempts, but the decision logic is the same: retry what could heal, fail fast on what cannot, stop when the budget is spent.

One more subtlety: a retry can create a duplicate. If the broker wrote the record but its answer was lost, the producer retries and the record is stored twice. Idempotence solves that, and it is the topic of Batching, Compression and Idempotence.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does KafkaProducer.send() return in the Java client?

    send() is asynchronous: it buffers the record and returns a Future<RecordMetadata>. You block on it, or use a callback, to learn the result.
  2. 2. Which error is NOT worth retrying?

    A record that is too large will be rejected every time. The other three are temporary conditions that usually heal.
  3. 3. Why should a producer be closed or flushed before the program exits?

    send() only buffers. close() and flush() wait until buffered records have been delivered, otherwise the last ones can be lost.
  4. 4. With acks=all, when does the leader answer the producer?

    acks=all waits for the in-sync replicas, which is the strongest durability setting a producer can ask for.

Summary

  • A producer builds records, serializes them, picks a partition, batches them and sends them to partition leaders.
  • send is asynchronous; use the returned future or a callback to learn about success or failure, and always flush or close before exit.
  • acks chooses between speed (0), leader-only durability (1) and in-sync-replica durability (all, the default in current clients).
  • Retriable errors are retried until delivery.timeout.ms; non-retriable errors fail immediately and need a decision from your code.
  • Retries can duplicate records unless idempotence is on.

Next, make producers fast and safe in Batching, Compression and Idempotence.