Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 39 ยท Topics and Partitions

Message Keys and Partitioners

See how Kafka picks a partition from a record key with murmur2, how the sticky partitioner handles keyless records, and how hot keys and custom partitioners work.

Every record you send to a multi-partition topic must land in exactly one partition. The rule that decides which one is the partitioner, and the main input to it is the record's key. Keys give you the one ordering guarantee Kafka offers: all records with the same key go to the same partition, so they are read in the order they were written.

This lesson shows what the default partitioner does for keyed and unkeyed records, what goes wrong with uneven keys (hot keys), and when to write your own partitioner.

What a key is for

A key is an optional byte array attached to a record, such as a customer ID, an order ID or a device ID. Choose it to be the thing whose events must stay in order. If you key order-changed events by order ID, then "created", "paid" and "shipped" for order 42 are always in the same partition and are consumed in that sequence.

How the default partitioner decides

Key present Hash the key bytes, then take the result modulo the partition count.
No key Pick a partition and stay with it for a batch.
Explicit partition If the producer names one, it is used.

Same key plus same partition count means the same partition.

1 / 2

For a keyed record, the Java client's default partitioner computes murmur2(keyBytes), forces the result to be non-negative, and takes it modulo the number of partitions. Murmur2 is a fast, non-cryptographic hash with a good spread, so many different keys land fairly evenly across partitions. Clients written for other languages generally implement the same idea, and many copy murmur2 for compatibility, but check your client's documentation if different languages must agree on placement.

Seeing the hash at work

The code below is a JavaScript port of the murmur2 function, followed by the "positive then modulo" step. The same key always gives the same partition, and many distinct keys spread out nearly evenly. Treat the exact partition numbers as illustrative; what matters is the behavior.

JS Key to partition with murmur2

Press Run (or Ctrl + Enter) to see the console output here.

No output.

user-1 appears twice and goes to the same partition both times, which is exactly the ordering promise. The second part shows an even spread: with 10,000 keys, each of the six partitions gets roughly a sixth. Notice that two different keys (user-1 and user-2) may share a partition; a partition holds many keys, but one key lives in only one partition.

Records without a key

If a record has no key, there is nothing to hash, and any partition is fine. Spreading each record to the next partition (pure round-robin) would work but creates tiny batches: with N partitions, a producer collecting records for a batch would fill N half-empty batches instead of one full one. Since Kafka 2.4 the producer instead uses a sticky approach: it sends records to one partition until a batch is full (or the batch is sent), then switches to another. Over time the load is still even, but batches are bigger, which means fewer requests, better compression and lower latency. Later releases refine this further, for example by favoring partitions on brokers that respond quickly.

The consequence is easy to miss: unkeyed records written close together will often sit in the same partition. If you compare output from unkeyed sends, do not expect strict one-by-one rotation.

Hot keys and skew

The default partitioner spreads keys evenly, not traffic. If one key is much busier than the rest, its partition becomes a hot spot: more disk, more network and a consumer that falls behind while others are idle. Typical causes are a single huge customer, a bot, or a default key such as "unknown" or null-turned-string.

The next stand-in builds traffic where one customer sends 60% of the orders. It prints how the default partitioner distributes that load, what happens if the hot key is salted (given a suffix so it spreads over several partitions), and how a custom rule can pin certain keys.

JS Skew, salting and a custom partitioner

Press Run (or Ctrl + Enter) to see the console output here.

No output.

In the default run, one partition holds roughly 70% of all records because big-shop is hashed to a single partition. Salting the hot key spreads that traffic over several partitions (the split is not perfectly even, since eight salt values hash into four partitions), at a price: the key's records are now spread over several partitions, so their order is gone. Salting trades ordering for balance, so use it only when per-key order does not matter or can be restored downstream.

The custom partitioner keeps vip- customers on partition 0 and hashes everyone else over the remaining partitions. The last lines contrast the sticky pattern (a batch's worth of records to each partition in turn) with strict round-robin.

Writing a custom partitioner

In the Java client, a partitioner implements the Partitioner interface and is set with the partitioner.class producer property. It receives the topic, key, value and the current cluster metadata, and returns a partition number:

Java
public class VipPartitioner implements Partitioner {
    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        int partitions = cluster.partitionsForTopic(topic).size();
        String id = String.valueOf(key);
        if (id.startsWith("vip-")) {
            return 0;
        }
        return 1 + Utils.toPositive(Utils.murmur2(keyBytes)) % (partitions - 1);
    }

    @Override public void configure(Map<String, ?> configs) { }
    @Override public void close() { }
}

This code is read-only here. Think carefully before writing a custom partitioner. A bug in it breaks ordering silently, and consumers know nothing about it, so the rule must stay the same for the life of the data. Most systems are better served by choosing a good key, or by salting only where required.

Ordering guarantees, summarized

  • Same key and same partition count means same partition, so per-key order is preserved.
  • Order across different keys is not guaranteed unless they happen to share a partition.
  • If a producer retries after a failure, order can change unless idempotence is on or max.in.flight.requests.per.connection is 1 (batching and idempotence).
  • Adding partitions changes where keys hash to (topics and partitions).

Pick the key by asking "which events must be processed in order relative to each other?" The answer (order ID, account ID, device ID) is usually your key. Check later that the key space is large and fairly even.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How does the default partitioner pick a partition for a record with a key?

    The same key bytes always hash to the same value, so with the same partition count the same key always maps to the same partition.
  2. 2. Why does the producer use a sticky approach for records without a key?

    Sending consecutive unkeyed records to one partition until a batch is full yields fewer, bigger requests. The load still evens out over time.
  3. 3. One customer produces 60% of all records and is used as the key. What is the likely problem?

    All records for that key go to one partition, so that partition (and its consumer) gets most of the load.
  4. 4. What do you give up by salting a hot key with random suffixes?

    Salting splits one key into several, which land in different partitions. Order exists only inside a partition, so the original key's order is lost.
  5. 5. What is a risk of a custom partitioner?

    Consumers rely on the placement rule staying stable. Changing or breaking it moves keys and breaks per-key ordering without any error.

Summary

  • A key decides the partition: hash the key (murmur2 in the Java client), make it positive, and take the modulo of the partition count.
  • Same key plus same partition count gives the same partition, which is how per-key order is kept.
  • Unkeyed records use a sticky strategy that fills batches before switching partitions.
  • Hot keys cause skew; salting spreads load but gives up per-key order.
  • Custom partitioners are possible via partitioner.class but must stay stable and correct.
  • The code here is a stand-in port; real producers do this inside the client library.

Next: Replication, Leaders and ISR, how Kafka keeps copies of each partition.