Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 39 ยท Projects

Project: Order Processing Pipeline

Build an order pipeline step by step: a keyed producer, a partitioned orders topic, a fulfilment consumer group with a crash, and idempotent handling, with runnable stand-ins.

Time to put the pieces together. In this project an online shop publishes every new order to Kafka. A fulfilment service, scaled to several workers, reads those orders and ships them. Along the way a worker crashes in the middle of its job, the group rebalances, and an order is delivered a second time. The pipeline must still ship every order exactly once, in the right order for each customer.

That small story touches nearly everything you have learned: keys and partitions, consumer groups, offsets and commits, at-least-once delivery and idempotent handlers.

Real versus stand-in. The topic creation command and the Java/JavaScript sketches below are real Kafka usage, shown read-only because the site cannot reach a cluster. The TryIt blocks are hand-written JavaScript stand-ins: an in-memory three-partition topic, a toy hash instead of Kafka's murmur2, and a simple group coordinator. They show the logic of the pipeline, not the real client.

The pipeline

Shop Publishes OrderPlaced, keyed by customer.
orders topic Three partitions, replicated.
Fulfilment group Workers share the partitions.
Warehouse Must receive each order once.
1 / 2

Step 1: design the topic and the event

The topic is called orders. Each record is an OrderPlaced fact (see Event-Driven Patterns):

JSON
{ "orderId": "o1001", "customer": "ana", "total": 35.0 }

Three decisions:

  • The key is the customer id. Orders of the same customer always go to the same partition and are read in the order they were placed. Orders of different customers have no ordering relationship, so they can be spread out and processed in parallel. Keying by orderId would spread load more evenly but lose per-customer order (see Message Keys and Partitioners).
  • Three partitions allow up to three workers in the group to work in parallel (see Topics and Partitions). Plan for growth, because adding partitions later changes which partition a key maps to.
  • Replication factor 3 and min.insync.replicas=2, with producers using acks=all, so the loss of one broker loses no orders (see Replication, Leaders and ISR).

Creating the topic on a real cluster looks like this:

Terminal
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders \
  --partitions 3 --replication-factor 3 \
  --config min.insync.replicas=2

(On a single-broker development machine, use --replication-factor 1 and drop the min.insync.replicas option.)

Step 2: the producer

A real Java producer for the shop would use the settings from Writing a Producer and Batching, Compression and Idempotence:

Java
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");
props.put("enable.idempotence", "true");

try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
    String json = "{\"orderId\":\"o1001\",\"customer\":\"ana\",\"total\":35.0}";
    producer.send(new ProducerRecord<>("orders", "ana", json), (meta, error) -> {
        if (error != null) {
            // log, alert, and decide: retry later, store locally, or reject the order
            System.err.println("order not published: " + error);
        }
    });
}

The stand-in below plays the shop. It publishes twelve orders from six customers and prints where each lands. Orders of one customer always share a partition.

JS Step 2: keyed producer

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Step 3: the fulfilment consumer group

Every worker runs the same program with the same group.id. The group coordinator hands each partition to exactly one worker (see Consumer Groups and Rebalancing). Three partitions means three workers at most; a fourth would sit idle.

A Java worker, simplified:

Java
props.put("group.id", "fulfilment");
props.put("enable.auto.commit", "false");
props.put("auto.offset.reset", "earliest");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(List.of("orders"));
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> record : records) {
            shipIfNew(record);          // idempotent, see below
        }
        consumer.commitSync();          // only after the whole batch is handled
    }
}

Auto-commit is off so the offset moves only after the work is done (see Offsets and Commits). That is at-least-once: if a worker dies after shipping but before committing, the next owner of the partition starts from the last committed offset and sees those orders again.

Step 4: make handling idempotent

Because duplicates are possible (see At-Most-Once, At-Least-Once, Exactly-Once), shipIfNew must be safe to run twice. In a real service the simplest reliable way is a database table of shipped order ids with a unique constraint, written in the same transaction as the shipment record:

SQL
CREATE TABLE shipments (
  order_id TEXT PRIMARY KEY,
  customer TEXT NOT NULL,
  shipped_at TIMESTAMP NOT NULL
);

-- inside the handler's transaction:
INSERT INTO shipments (order_id, customer, shipped_at)
VALUES (?, ?, CURRENT_TIMESTAMP)
ON CONFLICT (order_id) DO NOTHING;
-- if zero rows were inserted, this order was already handled: skip the warehouse call

If the warehouse call itself is not transactional with your database, give the warehouse the order id as an idempotency key, so a repeated request is recognized there too. External side effects are the limit of any Kafka guarantee.

Step 5: run the whole pipeline

The next stand-in is the complete system in one program: the producer from step 2, a consumer group with two workers, manual commits after processing, and an idempotent handler. In round 2 worker w2 crashes after shipping an order but before committing its offset. The group rebalances, w1 takes over w2's partition, and re-reads the uncommitted order. The handler recognizes the duplicate.

JS Step 5: end-to-end pipeline with a crash

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Read the output from the top. In round 1 the group has two workers and w1 owns two partitions, w2 one. In round 2 w2 ships o1008 and crashes before committing. Round 3 starts with a new assignment where w1 owns all three partitions, begins p1 at the last committed offset, sees o1008 again and skips it. Twelve orders are shipped from thirteen handler calls, and each customer's orders appear in the order they were placed.

Try changing the experiment: set crashed logic to crash w1 instead, or move the commit above the handle(rec) call. With the commit before the work, a crash loses an order instead of repeating it, which is the at-most-once trade-off. And remove the shipped check to see the duplicate shipment appear.

Step 6: failure handling and next steps

A production version needs a few more pieces.

  • Poison orders. A malformed record must not block its partition forever. Route it to a dead letter topic as described in Retries, Dead Letter Topics and Poison Messages.
  • Lag alerts. Alert when the fulfilment group's lag keeps growing (see Monitoring, Lag and Troubleshooting).
  • Atomic publishing. If the shop saves orders in a database, publish them with the outbox pattern rather than writing to both separately.
  • Schemas. Register a schema for OrderPlaced so producers and consumers can evolve independently (see Schemas, Avro, Protobuf and JSON).
  • Security. Give the shop write access to orders and the fulfilment service read access to orders and its group, nothing more (see Security and Quotas).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is the order's customer id used as the record key?

    The partitioner maps equal keys to the same partition, and a partition preserves order. Different customers can still be processed in parallel.
  2. 2. In the crash scenario, why is the order shipped only once even though it was delivered twice?

    The consumer commits after processing, so redelivery after a crash is possible. The idempotent handler turns the second delivery into a no-op.
  3. 3. How many workers of the fulfilment group can be busy at the same time with a three-partition topic?

    A partition is read by one member of a group at a time, so the partition count is the ceiling for parallelism.
  4. 4. What would happen if the worker committed the offset before shipping the order, and then crashed?

    Committing first gives at-most-once behavior: a crash after the commit but before the work skips that record for good.

Summary

  • Key records by the entity whose order matters (here the customer) and size partitions for the parallelism you need.
  • Use acks=all, replication and min.insync.replicas so accepted orders survive a broker failure.
  • Commit offsets after the work, accept at-least-once, and make the handler idempotent so duplicates are harmless.
  • A crash makes the group rebalance; the new owner resumes from the last committed offset.
  • A production pipeline adds a dead letter topic, lag alerts, schemas, security and an outbox for atomic publishing.

Next, build a streaming pipeline with windows in Project: Real-Time Page-View Analytics.