Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 39 ยท Getting Started

Your First Topic with the CLI

Start a single-node KRaft Kafka broker, create a topic with kafka-topics, and send and read messages with the console producer and consumer.

The fastest way to feel how Kafka works is to run a broker on your own machine and talk to it with the command-line tools that ship with Kafka. In this lesson you start one broker, create a topic, type a few messages into a console producer and read them back with a console consumer.

The commands below are read-only on this site: a real broker needs a server and a Java runtime, so you run them in your own terminal. After each group of commands, a runnable stand-in shows the same session so you can see what each part does and what its output means.

Commands are shown for Kafka 4.x, which runs only in KRaft mode (no ZooKeeper). If you use a 3.x release, the config file lives at config/kraft/server.properties instead of config/server.properties. Output formats vary a little between versions.

Get Kafka running

You need a Java runtime (Kafka 4.x needs Java 17 or newer for the broker). Download a Kafka release from the Apache Kafka site, extract it and open a terminal in the extracted folder.

A KRaft broker needs a cluster ID and a formatted storage directory before it first starts. For a single local broker:

Terminal
# 1. Create a random cluster ID
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

# 2. Format the log directory (Kafka 4.x adds --standalone for a single-node setup)
bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties

# 3. Start the broker (keeps running in this terminal)
bin/kafka-server-start.sh config/server.properties

On Kafka 3.x, step 2 is the same command without --standalone, using the config/kraft/server.properties file. When the log says the broker has started, it is listening on localhost:9092. Leave this terminal open and use a second terminal for the following commands.

If you prefer Docker, the Apache Kafka project publishes an image that already contains a single-node KRaft broker:

Terminal
docker run -d --name broker -p 9092:9092 apache/kafka:latest

# run the CLI tools inside the container
docker exec -it broker /opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

The --bootstrap-server option is how every tool finds the cluster: you give it one or more broker addresses, and the tool learns about the rest from them.

Create and inspect a topic

The kafka-topics.sh tool manages topics.

Terminal
# create a topic with 3 partitions
bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

# list topics
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

# show details
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

A replication factor of 1 means a single copy, which is all one broker can offer. Real clusters usually use 3 (replication and ISR). If you omit --partitions, the broker's num.partitions default applies, which is 1.

The --describe output has one summary line and one line per partition. Each partition line shows the Leader (the broker handling reads and writes for that partition), the Replicas (every broker holding a copy) and the Isr (the replicas that are in sync). With one broker, all three show broker 1.

Try the stand-in, which understands a tiny part of kafka-topics.sh. Its messages are simplified compared with the real tool.

JS Stand-in: kafka-topics.sh

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that creating orders twice fails and that payments gets one partition when you do not say otherwise. The real tool also fails on a duplicate topic, with its own wording. (Brokers have auto.create.topics.enable on by default, so a producer writing to a missing topic can create it silently; many teams switch that off in production and create topics deliberately.)

Send and read messages

Open one terminal for the consumer and another for the producer. The console producer turns every line you type into a record:

Terminal
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic greetings
> hello
> kafka
> world

Press Ctrl+C to stop. The console consumer prints records as they arrive:

Terminal
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic greetings --from-beginning
hello
kafka
world

Without --from-beginning, the console consumer starts at the end of the topic and shows only records that arrive after it starts. That matches the default of a consumer with no saved position (auto.offset.reset=latest), which is explained in reading with a consumer.

To send keys, tell the producer how to split each line, and ask the consumer to print them:

Terminal
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic greetings \
  --property parse.key=true --property key.separator=:
> ada:hello
> bob:hi

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic greetings \
  --from-beginning --property print.key=true

Keys decide which partition a record goes to, which you will study in message keys and partitioners.

The second stand-in plays a producer session and three consumer sessions on a single-partition topic. It prints the same shape of session you would see in a terminal.

JS Stand-in: console producer and consumer

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The first consumer sees nothing because the records were written before it started and it began at the end. The --from-beginning consumers read all three. When a record has no key, the key prints as null. In a real session, the consumer keeps waiting for new records until you press Ctrl+C.

Installing on Windows, macOS and Linux

The download is the same everywhere: a .tgz archive from the Apache Kafka site. Only the way you open a shell and start the scripts changes.

  • Linux: install a JDK 17+ with your package manager (for example sudo apt install openjdk-21-jdk), extract the archive with tar -xzf kafka_*.tgz and run the bin/*.sh scripts exactly as shown above.
  • macOS: brew install openjdk@21 (or any JDK 17+), then the same tar and bin/*.sh commands. Homebrew also has a kafka formula, but the archive keeps every version side by side and matches the documentation.
  • Windows: the supported way is WSL 2 (Windows Subsystem for Linux). Open an Ubuntu terminal, follow the Linux steps and keep the Kafka folder inside the Linux file system (~/kafka), not under /mnt/c, which is much slower. Current Kafka releases no longer ship bin\windows\*.bat scripts, so running Kafka natively in PowerShell is not recommended. Docker Desktop (below) is the other easy route. From Windows programs, localhost:9092 reaches a broker running in WSL 2 or Docker Desktop.

Check the setup with java -version, then bin/kafka-topics.sh --version.

A three-broker cluster with Docker Compose

A single broker cannot show replication. This Compose file starts three KRaft nodes that act as both broker and controller (the "combined" mode, fine for learning, not for production; see Brokers, Controllers and KRaft), plus the open-source Kafka UI web interface.

YAML
x-kafka-common: &kafka-common
  image: apache/kafka:latest
  environment: &kafka-env
    KAFKA_PROCESS_ROLES: broker,controller
    KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
    KAFKA_LISTENERS: INTERNAL://:19092,EXTERNAL://:9092,CONTROLLER://:9093
    KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
    KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
    KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
    KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
    KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
    KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2

services:
  kafka-1:
    <<: *kafka-common
    ports: ["9092:9092"]
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 1
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-1:19092,EXTERNAL://localhost:9092
  kafka-2:
    <<: *kafka-common
    ports: ["9094:9092"]
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 2
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-2:19092,EXTERNAL://localhost:9094
  kafka-3:
    <<: *kafka-common
    ports: ["9095:9092"]
    environment:
      <<: *kafka-env
      KAFKA_NODE_ID: 3
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-3:19092,EXTERNAL://localhost:9095
  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    ports: ["8080:8080"]
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka-1:19092,kafka-2:19092,kafka-3:19092
    depends_on: [kafka-1, kafka-2, kafka-3]

Run docker compose up -d, open http://localhost:8080 to browse topics, messages and consumer groups, and create a topic with --replication-factor 3 to watch replicas spread over the nodes. The two listeners matter: containers talk to each other on INTERNAL, while programs on your machine use the EXTERNAL addresses (localhost:9092, localhost:9094, localhost:9095). Exact environment variable names can differ between image versions, so check the image documentation if a node refuses to start. The apache/kafka image has no kafka-storage format step for you to run; it formats storage on first start.

Common startup problems

JS Which startup error is this?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A note on ZooKeeper

Older tutorials start zookeeper-server-start.sh first. ZooKeeper mode was removed in Kafka 4.0; every current release uses KRaft, so skip those instructions. If you must run an old 3.x cluster, migrate it to KRaft on the latest 3.x release before upgrading to 4.x (see Brokers, Controllers and KRaft for how KRaft works).

Other tools worth knowing

Kafka ships more scripts in the bin folder. You will use these in later lessons:

  • kafka-consumer-groups.sh shows consumer groups, their positions and lag (monitoring and lag).
  • kafka-configs.sh reads and changes topic and broker configuration.
  • kafka-acls.sh manages permissions (security and quotas).
  • kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh measure throughput.

Every script prints its options when you run it with --help.

The console tools are for learning and quick checks. They are not a good basis for real applications, which use a client library with proper error handling, serialization and offset management.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the --bootstrap-server option tell a Kafka tool?

    The tool connects to the given broker and learns about the other brokers and partition leaders from it. You can list more than one address for resilience.
  2. 2. You start kafka-console-consumer.sh without --from-beginning on a topic that already holds 100 records. What do you see?

    With no saved position the consumer starts at the end of the log, so it prints only new records. Add --from-beginning to read the history.
  3. 3. In the output of --describe, which field names the broker that handles reads and writes for a partition?

    Each partition has one leader broker. Replicas lists all brokers that hold a copy and Isr lists those that are in sync.
  4. 4. Why can you only run this lesson's real commands in your own terminal?

    A real broker is a long-running server that needs Java and disk storage. The stand-ins on this page model the sessions instead.

Summary

  • A KRaft broker needs a cluster ID and formatted storage, then starts with kafka-server-start.sh; Docker offers an apache/kafka image.
  • kafka-topics.sh creates, lists and describes topics; --describe shows leader, replicas and ISR per partition.
  • The console producer sends each line as a record; the console consumer reads from the end unless --from-beginning is given.
  • Keys can be sent with parse.key and shown with print.key.
  • The commands are read-only here; the stand-ins imitate the sessions with plain JavaScript.

Next: Topics and Partitions, how a topic is split into parallel logs.