Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 39 ยท Getting Started

What Is Apache Kafka?

Learn what Apache Kafka is, how brokers, topics, producers and consumers fit together, where Kafka shines, and when a plain queue or database is better.

Apache Kafka is a platform for moving and storing streams of events. An event is a small fact that something happened: a customer placed an order, a sensor reported a temperature, a user clicked a button. Kafka lets many programs write those facts down in order, keeps them for as long as you decide, and lets many other programs read them, each at its own pace.

Kafka started at LinkedIn to move huge amounts of activity data between systems, and it is now an Apache project used for log pipelines, real-time analytics, microservice communication and data integration. This lesson gives you the big picture and the vocabulary. The rest of the tutorial fills in every part.

Three things people compare Kafka to

Kafka is often described as "a message queue", "a database" or "a log". Each description is partly right, so it helps to see how it differs from the first two.

Kafka sits between familiar tools

Message queue Moves messages, but removes them once read.
Database Stores current state and answers queries.
Event log Stores what happened, in order, for replay.

Kafka is closest to the log, with queue-like and database-like powers.

1 / 2

A classic queue hands each message to one worker and then forgets it. If a second system wants the same message, you have to copy it into a second queue. Kafka does not delete a message when it is read, so a billing service, a search indexer and an analytics job can all read the same stream without knowing about each other. A new service can even join next year and read last month's events from the start.

A database keeps the latest state ("this customer's address is X"). Kafka keeps the history of changes ("the address was set to X, then to Y"). You can rebuild state from history, but Kafka is not built for ad-hoc queries such as "find all customers in Berlin". Typically you use Kafka to feed databases, not to replace them.

The parts of Kafka

Here are the five words you need first.

  • A record (also called an event or message) has an optional key, a value, a timestamp and optional headers.
  • A topic is a named stream of records, such as orders or page-views.
  • A broker is a Kafka server. It stores topic data on disk and serves clients. A cluster is several brokers working together.
  • A producer is a program that writes records to a topic.
  • A consumer is a program that reads records from a topic. Consumers that cooperate form a consumer group.
flowchart LR
  P1["Producer: web shop"] --> T
  P2["Producer: mobile app"] --> T
  subgraph K["Kafka cluster"]
    T["Topic: orders<br/>(stored on brokers)"]
  end
  T --> C1["Consumer: billing"]
  T --> C2["Consumer: analytics"]
  T --> C3["Consumer: email service"]

Producers append records to a topic; each consumer reads the topic independently.

Two more ideas appear soon. Each topic is split into partitions so it can spread over many brokers (topics and partitions), and each record in a partition gets a number called its offset (the commit log idea). Producers and consumers are decoupled: a producer never waits for a consumer, and a slow consumer never slows down a producer.

What Kafka is used for

Typical jobs for Kafka include:

  • Activity tracking. Page views, searches and clicks from a website flow into a topic, and several teams read them.
  • Service integration. An order-placed event lets payment, stock and shipping services react without calling each other directly.
  • Log and metric pipelines. Servers write their logs into Kafka, and tools move them on to storage and dashboards.
  • Stream processing. Programs compute running totals, detect fraud patterns or join two streams in real time (Kafka Streams).
  • Change data capture. Changes in a database table are published as events so that caches and search indexes stay in sync.
  • Event sourcing. The log of events is the source of truth, and current state is derived from it.

Kafka as a pub-sub system and for log aggregation

In a publish-subscribe system, publishers do not address receivers. They publish to a named channel, and every interested subscriber gets its own copy. Kafka is pub-sub where each subscriber is a consumer group with its own position in the log: two groups reading orders each see every record, while the members inside one group share the work. Unlike classic pub-sub brokers, Kafka keeps the messages after delivery, so a new subscriber can start from the beginning.

Log aggregation is the original use case. Without Kafka, every application server writes log files, and a collector must be wired to each one. With Kafka, each server (or a small agent beside it) publishes log lines to a topic such as app-logs. One stream then feeds storage (object storage or a search index), alerting and dashboards, and a new consumer can be added without touching the servers. Because the topic is partitioned by service or host, ordering within one source is preserved.

How the pieces interact

  1. Producers publish records to a topic; the partitioner picks the partition.
  2. The broker that leads that partition appends the record to its log and copies it to the follower replicas.
  3. Consumers in a group pull records from the partitions assigned to them and commit their offsets.
  4. The cluster controller (KRaft) tracks brokers, leaders and topic metadata (cluster and KRaft).

Topic-level settings

Each topic can override broker defaults. The common ones are retention.ms, retention.bytes, cleanup.policy (delete or compact), min.insync.replicas, max.message.bytes and segment.bytes. Change them on a live topic with the config tool:

Terminal
kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --entity-type topics --entity-name orders \
  --add-config retention.ms=86400000,min.insync.replicas=2

Why this architecture works well

  • Scalability: add partitions and brokers; consumers scale by adding group members.
  • Durability: replicated, on-disk logs survive broker failures.
  • Throughput: sequential disk writes, batching and compression.
  • Decoupling: producers and consumers never talk to each other and can fail or restart independently.
  • Replay: consumers can rewind to reprocess history.

When Kafka is the wrong tool

Kafka is powerful but not small. It adds a cluster to run and a lot of concepts to learn. Consider something simpler when:

  • You have low volume and one consumer. A database table or a plain job queue is easier.
  • You need per-message features such as priorities, delayed delivery or complex routing. Classic brokers like RabbitMQ are built around those.
  • You need request/response calls with an immediate answer. Use an ordinary HTTP or RPC call.
  • You need to query data by arbitrary fields. Use a database or a search engine, and let Kafka feed it.
  • You cannot afford the operations work. Managed Kafka services exist, but they still cost money and require design effort.

For the general ideas behind queues and publish/subscribe, see queues and pub/sub.

Try it: queue versus log

Real Kafka cannot run inside this page, so the examples in this tutorial are small stand-ins: plain JavaScript that models the idea. The real broker is a separate server you install; everything it does is more complex, but the core behavior is the same. The first stand-in shows the main difference between a queue and a log.

JS A queue forgets, a log remembers

Press Run (or Ctrl + Enter) to see the console output here.

No output.

In the queue, once two workers have taken messages, a late reader can only see what is left. In the log, billing is two records in while analytics is one record in, and the data is still complete. That "each reader has its own position" idea is the heart of Kafka.

Try it: a tiny publish/subscribe broker

Now a slightly bigger stand-in. MiniBroker has topics (arrays of records) and remembers, for each group and topic, which offset to read next. Producing returns the offset of the new record.

JS MiniBroker: topics, offsets and independent readers

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that the search-index group first reads only two records (max = 2), then catches up later, while analytics already finished. Neither group changed what the other sees. The last call returns nothing because analytics has read everything so far.

What is real here and what is not: topics, records, keys, offsets and independent group positions are real Kafka concepts. The real broker, though, stores records in files on disk, splits topics into partitions, replicates them to other brokers and tracks group positions inside Kafka itself.

Kafka usually pushes you toward asynchronous designs. A producer's job is done when the broker has safely stored the record, not when everyone has processed it. That makes systems more resilient, but also means you must think about duplicates, ordering and lag, which later lessons cover.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens to a record in a Kafka topic after a consumer reads it?

    Reading does not delete records. They stay until the topic's retention settings remove them, which is why many consumers can read the same data.
  2. 2. Which statement about producers and consumers is true?

    Both sides talk only to the brokers. A slow consumer does not slow a producer down, and a topic can have many independent consumers.
  3. 3. Which need is a good reason NOT to use Kafka?

    A single low-volume queue is easier to run with a database table or a simple job queue. Kafka's strengths show with many readers, replay and high volume.
  4. 4. What is a broker?

    Brokers are the servers that make up a Kafka cluster. They store the log data on disk and answer requests from producers and consumers.

Summary

  • Kafka stores streams of events in topics and keeps them after they are read.
  • Producers append records; consumers read at their own pace, each keeping its own position (offset).
  • Brokers are the servers; a cluster of brokers shares and replicates the data.
  • It fits activity tracking, service integration, log pipelines, stream processing and change data capture.
  • It is a poor fit for tiny single-consumer jobs, per-message routing features, request/response calls and ad-hoc queries.
  • The runnable examples in this tutorial are stand-ins that model Kafka's behavior; the real broker runs as its own server.

Next: The Commit Log Idea, the simple data structure that explains why Kafka behaves the way it does.