Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 39 ยท Topics and Partitions

Topics and Partitions

Learn how Kafka splits a topic into partitions, why ordering holds only inside one partition, how partition count limits parallelism, and what adding partitions does.

A single log on a single machine has limits: one disk, one network card, one reader at a time. Kafka removes those limits by splitting every topic into partitions. Each partition is its own independent log, and the partitions of a topic can live on different brokers and be read by different consumers at the same time.

Partitions are the unit of almost everything in Kafka: storage, replication, ordering and parallelism. Getting the partition design right is one of the most important decisions you make, because some choices (like the count) are awkward to change later.

A topic is a set of logs

If a topic is a named stream, a partition is one lane of that stream. The topic orders with three partitions is three separate logs: orders-0, orders-1 and orders-2. Each has its own offsets starting at 0. A record is identified by the triple topic, partition, offset.

A topic is made of partitions

Independent logs Each partition has its own offsets.
Spread out Partitions can sit on different brokers.
Parallel Different consumers read different partitions.

A record is addressed by topic, partition and offset.

1 / 2

When a producer sends a record, something must decide which partition gets it. If the record has a key, the key decides (the next lesson explains how). If it has no key, the producer spreads records across partitions. Either way, each record lands in exactly one partition.

Ordering lives inside a partition

Kafka guarantees that records within one partition are stored and read in the order they were appended. It does not guarantee any order between partitions. If you need all events for one customer in order, they must all go to the same partition, which is what message keys are for.

The stand-in below creates a topic with three partitions and sends nine unkeyed records. It uses simple round-robin placement (real producers are a bit smarter, see the next lesson). Then it reads all partitions the way a consumer would, polling them in a seeded pseudo-random order to imitate network timing.

JS A three-partition topic

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Compare the read order with the order the records were sent. e1 was sent first, yet e2 may be read before it. Check any single partition, though: e1 e4 e7 always appear in that relative order. A consumer that needs a total order across the whole topic cannot get it from Kafka; the usual answer is to design so that only per-key (and therefore per-partition) order matters.

If you truly need one global order, use a single partition. That is simple and correct, but it limits throughput to what one broker, one disk and one consumer can handle.

Partitions and parallelism

Within a consumer group, Kafka gives each partition to at most one consumer at a time. This is what makes ordered processing possible: nobody else is reading that partition at the same moment. It also means the partition count is the upper limit of parallelism for a consumer group. With four partitions, at most four consumers in the same group are busy; a fifth sits idle.

The same is true on the write side in practice: more partitions let producers spread load across more brokers.

JS How many consumers can work in parallel?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With four partitions, two consumers get two each, four consumers get one each, and six consumers leave two idle. (Real group assignment uses configurable strategies and rebalancing; see consumer groups.)

The second half of the output shows a hazard. If the key-to-partition rule is "hash modulo the number of partitions", changing the number of partitions moves most keys to a different partition. The hash here is a toy; real Kafka uses murmur2, but the arithmetic problem is the same.

Choosing the number of partitions

There is no universal number. Think about these trade-offs.

More partitions give you:

  • Higher possible consumer parallelism.
  • Higher write and read throughput by spreading load over brokers and disks.
  • Finer-grained data movement when you add brokers.

More partitions cost you:

  • More open files and more memory on brokers; every partition replica is a directory of files.
  • Longer recovery after a broker failure, because leaders for more partitions must be re-elected.
  • More replication traffic and more metadata for the cluster to manage.
  • Weaker ordering at the topic level, because order exists only inside each partition.
  • Higher end-to-end latency in some setups, since each partition batches separately.

A practical way to estimate: measure the throughput one partition can handle on the producer side and on the consumer side, divide your target throughput by each, and take the larger number. Then add some headroom for growth, because changing it later is harder. Many teams start with a modest count such as 6, 12 or 30 for busy topics, and fewer for small ones.

Adding partitions later

You can increase the number of partitions of an existing topic, but you cannot decrease it:

Terminal
bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --alter --topic orders --partitions 12

Existing records stay where they are; Kafka does not redistribute old data. New records for a key may now go to a different partition than older records for the same key, which breaks per-key ordering across the change and can confuse consumers that keep per-key state. If you rely on keys, plan the count up front, or create a new topic with more partitions and migrate to it.

Never treat the partition count as a casual tuning knob on keyed topics. Increasing it changes where keys land, and reducing it is not supported at all. The only way down is a new topic.

What is real and what is a stand-in

The Topic class here keeps partitions in arrays and chooses a partition by round-robin. A real producer chooses the partition in the client library (hash for keys, a sticky strategy for unkeyed records), and each partition is a replicated log on brokers. The concepts shown are real: partitions with their own offsets, ordering per partition only, and at most one consumer per partition within a group.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A topic has 6 partitions and a consumer group has 8 consumers. How many consumers receive data?

    Each partition is assigned to at most one consumer in a group. Six partitions can keep at most six consumers busy, and the other two stay idle.
  2. 2. Where does Kafka guarantee that records are read in the order they were written?

    Order is guaranteed per partition only. Records in different partitions can be read in any interleaving.
  3. 3. What is true about changing the partition count of an existing topic?

    Partitions can be added but never removed. Adding them also changes where keys hash to, so per-key placement changes for new records.
  4. 4. Which is a cost of having very many partitions?

    Each partition replica uses files, memory and leader-election work. Ordering inside a partition stays intact, and parallelism improves rather than shrinks.
  5. 5. Which three values identify one record in Kafka?

    Offsets are numbered per partition, so you need the topic, the partition and the offset to point at one record.

Summary

  • A topic is split into partitions; each partition is an independent, ordered, replicated log with its own offsets.
  • Order is guaranteed only inside a partition. Use keys to send related records to the same partition.
  • Within a consumer group each partition has one consumer, so the partition count caps parallelism.
  • More partitions raise throughput and parallelism but add files, metadata and recovery time.
  • Partitions can be added later but not removed, and adding them moves keys; plan the count up front.

Next: Message Keys and Partitioners, how a record's key picks its partition.