Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 39 ยท Operations

The AdminClient API and Topic Management

Manage Kafka topics from code and CLI: create, describe, delete, add partitions, alter configs, consumer-group admin and the rules that make some changes irreversible.

Topics are not only created by hand with a command. Applications, deployment tools and operators create them, change them and inspect them all the time. Kafka provides one API for this, the AdminClient, and the kafka-topics.sh, kafka-configs.sh and kafka-consumer-groups.sh tools are thin wrappers over it. This lesson covers what the AdminClient can do, and, just as important, what cannot be undone.

Real versus stand-in. The Java code and shell commands are real, shown read-only because the site cannot reach a Kafka cluster. The TryIt blocks are plain JavaScript stand-ins that imitate a cluster's topic metadata and enforce the same rules (for example that partitions can only grow). They are not the real client.

Creating the client

Java
Properties props = new Properties();
props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

try (Admin admin = Admin.create(props)) {
    // use admin here
}

Every call returns a result object holding KafkaFutures. Nothing has finished until you call .get() on them (or .all().get()), and errors arrive as ExecutionExceptions wrapping something like TopicExistsException or UnknownTopicOrPartitionException.

Create and describe topics

Java
NewTopic orders = new NewTopic("orders", 6, (short) 3)
    .configs(Map.of("retention.ms", "604800000", "min.insync.replicas", "2"));

admin.createTopics(List.of(orders)).all().get();

TopicDescription d = admin.describeTopics(List.of("orders"))
    .allTopicNames().get().get("orders");
d.partitions().forEach(p ->
    System.out.println(p.partition() + " leader=" + p.leader().id() + " isr=" + p.isr()));

The CLI equivalents:

Terminal
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders \
  --partitions 6 --replication-factor 3 --config retention.ms=604800000
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

The replication factor cannot exceed the number of live brokers, or creation fails. describe shows each partition's leader, replicas and in-sync replicas, the same information explained in Replication, Leaders and ISR.

Adding partitions, never removing them

Java
admin.createPartitions(Map.of("orders", NewPartitions.increaseTo(12))).all().get();
Terminal
bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic orders --partitions 12

Two rules:

  1. You can only increase. Kafka has no way to shrink a topic's partition count. To go down you create a new topic and copy the data across.
  2. Adding partitions changes key routing. The partitioner computes hash(key) % partitionCount, so the same key can start landing in a different partition. Old records stay where they were. If you rely on per-key ordering (see Message Keys and Partitioners), plan the partition count up front, or accept a one-time reordering at the moment you grow.

Changing the replication factor of an existing topic is not an AdminClient one-liner: you reassign replicas (kafka-reassign-partitions.sh or alterPartitionReassignments).

Altering configuration

Topic settings such as retention and compaction (see Retention and Segments) can change while the topic is live. Use incrementalAlterConfigs, which changes only the keys you name:

Java
ConfigResource topic = new ConfigResource(ConfigResource.Type.TOPIC, "orders");
AlterConfigOp setRetention =
    new AlterConfigOp(new ConfigEntry("retention.ms", "259200000"), AlterConfigOp.OpType.SET);
AlterConfigOp dropOverride =
    new AlterConfigOp(new ConfigEntry("max.message.bytes", ""), AlterConfigOp.OpType.DELETE);

admin.incrementalAlterConfigs(Map.of(topic, List.of(setRetention, dropOverride))).all().get();
Terminal
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics \
  --entity-name orders --alter --add-config retention.ms=259200000

DELETE removes the topic-level override so the broker default applies again. The old alterConfigs method replaced the whole config set and is deprecated; avoid it.

Stand-in: a topic registry with the real rules

JS Create, grow and alter topics

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Deleting topics

Java
admin.deleteTopics(List.of("orders-test")).all().get();
Terminal
bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic orders-test

Deletion is permanent: the log segments are removed from every replica. It is asynchronous, so the topic can briefly still appear in listings. Brokers have delete.topic.enable (default true), and managed clusters often restrict it with ACLs (Security and Quotas). Deleting a topic that consumers are still subscribed to makes them fail or, with auto-creation enabled, silently recreate it. Turn auto.create.topics.enable off in production so a typo in a topic name does not create a new topic.

Consumer-group administration

The same client works on groups:

Java
admin.listConsumerGroups().all().get();

Map<TopicPartition, OffsetAndMetadata> offsets =
    admin.listConsumerGroupOffsets("order-service")
         .partitionsToOffsetAndMetadata().get();

// move a stopped group back to a position
admin.alterConsumerGroupOffsets("order-service",
    Map.of(new TopicPartition("orders", 0), new OffsetAndMetadata(0))).all().get();
Terminal
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-service
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group order-service \
  --topic orders --reset-offsets --to-earliest --execute

Offsets can only be changed for a group that has no active members. Stop the consumers first, reset, then restart, which makes the group replay history (Offsets and Commits, Monitoring and Consumer Lag). Use --dry-run before --execute to see the result without applying it.

Stand-in: resetting a group's offsets

JS Reset offsets only when the group is stopped

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Practical advice

  • Treat topic definitions as code: keep them in version control and create them from a pipeline, not by hand.
  • Create the AdminClient once and reuse it; it is thread-safe.
  • Set timeouts on .get(30, TimeUnit.SECONDS) so a down cluster does not hang a deployment.
  • Prefer idempotent scripts: catch TopicExistsException and carry on.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which change to an existing topic is impossible?

    Kafka cannot shrink the number of partitions. You create a new topic and copy the data instead.
  2. 2. Why can adding partitions break per-key ordering?

    New records for a key can go to a different partition than its older records, so ordering across the change is lost.
  3. 3. When can a consumer group's committed offsets be reset?

    An active member would overwrite the new offsets, so Kafka requires an empty group.
  4. 4. What does admin.createTopics(...) return?

    AdminClient calls are asynchronous and return KafkaFutures; call get() or all().get() to wait and to see errors.

Summary

  • The AdminClient (and the kafka-topics, kafka-configs, kafka-consumer-groups tools) creates, describes, alters and deletes topics and manages groups.
  • Partitions can only be increased, and doing so changes which partition a key maps to.
  • Use incrementalAlterConfigs to change topic settings live; DELETE restores the broker default.
  • Topic deletion is permanent; disable auto-creation in production.
  • Group offsets can be reset only while the group is stopped.

Next: keep an eye on the cluster you manage with Monitoring and Consumer Lag.