Skip to content
Tutorials Center
My learning

Topics

Apache Kafka Tutorial

Learn Apache Kafka step by step: 39 lessons in 11 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Getting Started

  1. 1 What Is Apache Kafka? Learn what Apache Kafka is, how brokers, topics, producers and consumers fit together, where Kafka shines, and when a plain queue or database is better.
  2. 2 Kafka vs RabbitMQ, ActiveMQ, Pulsar and Cloud Queues Compare Kafka with RabbitMQ, ActiveMQ, Apache Pulsar, Amazon SQS, Google Pub/Sub and Azure Service Bus: queues versus logs, push versus pull, replay, routing and when to choose each.
  3. 3 The Commit Log Idea Understand the append-only commit log behind Kafka: offsets, immutability, replay, retention and why sequential disk writes make Kafka fast.
  4. 4 Your First Topic with the CLI Start a single-node KRaft Kafka broker, create a topic with kafka-topics, and send and read messages with the console producer and consumer.

Topics and Partitions

  1. 5 Topics and Partitions Learn how Kafka splits a topic into partitions, why ordering holds only inside one partition, how partition count limits parallelism, and what adding partitions does.
  2. 6 Message Keys and Partitioners See how Kafka picks a partition from a record key with murmur2, how the sticky partitioner handles keyless records, and how hot keys and custom partitioners work.
  3. 7 Replication, Leaders and ISR Understand Kafka replication: leaders and followers, the in-sync replica set, acks and min.insync.replicas, the high watermark, and what unclean election risks.

Producers

  1. 8 Writing a Producer Write a Kafka producer in Java and JavaScript: records, serializers, send(), callbacks, acks, retriable errors and delivery timeouts, with a runnable producer stand-in.
  2. 9 Batching, Compression and Idempotence Tune Kafka producer throughput with linger.ms, batch.size and compression, then prevent duplicates with enable.idempotence and sequence numbers. Runnable stand-ins included.

Consumers

  1. 10 Reading with a Consumer Learn the Kafka consumer poll loop: subscribe vs assign, deserializers, auto.offset.reset, max.poll.records and graceful shutdown, with a runnable poll-loop stand-in.
  2. 11 Consumer Groups and Rebalancing Understand Kafka consumer groups: partition assignment, range vs round-robin vs cooperative-sticky, heartbeats, session timeout, static membership and scaling limits.
  3. 12 Offsets and Commits Learn how Kafka consumers track progress: committed offsets, __consumer_offsets, auto vs manual and sync vs async commits, seek and replay, and consumer lag, with runnable stand-ins.

Delivery Guarantees

  1. 13 At-Most-Once, At-Least-Once, Exactly-Once Where Kafka duplicates and data loss come from, how commit ordering gives at-most-once or at-least-once, and how idempotent consumers and dedup keys approach exactly-once.
  2. 14 Transactions and Exactly-Once Understand Kafka transactions: transactional.id, read_committed, consume-transform-produce, zombie fencing and where exactly-once stops, with runnable stand-ins.

Storage: Retention and Compaction

  1. 15 Retention, Segments and Disk Layout See how Kafka stores partitions as segment files with indexes, how time and size retention delete old data, and what tiered storage changes, with runnable stand-ins.
  2. 16 Log Compaction and Tombstones Learn how Kafka log compaction keeps the latest value per key, how tombstones delete keys, and how compacted topics act as tables, with a runnable stand-in.

Data Formats and the Ecosystem

  1. 17 Schemas, Avro, Protobuf and JSON Why Kafka topics need schemas, how a schema registry works, what backward, forward and full compatibility mean, and how to evolve a schema safely, with runnable stand-ins.
  2. 18 Kafka Connect Learn Kafka Connect: source and sink connectors, workers, tasks, converters, single message transforms, offsets and the REST API, with a runnable pipeline stand-in.
  3. 19 Kafka Streams and Stream Processing Learn Kafka Streams and stream processing: KStream vs KTable, stateless and stateful operations, windowing, joins and state stores with changelog topics, with runnable stand-ins.
  4. 20 Kafka from Python Use Kafka from Python with confluent-kafka: produce, delivery callbacks, flush, poll loops, group.id, auto.offset.reset, SSL/SASL settings, and kafka-python compared.
  5. 21 ksqlDB: SQL over Kafka Process Kafka topics with SQL in ksqlDB: streams versus tables, persistent, push and pull queries, windowed aggregates and joins, with runnable stand-ins.
  6. 22 Kafka with Spark Structured Streaming and Flink Read and write Kafka topics from Spark Structured Streaming and Flink: source and sink options, offsets and checkpoints, exactly-once sinks, with runnable stand-ins.

Operations

  1. 23 Running a Cluster: KRaft, Brokers and Controllers Understand how a Kafka cluster runs without ZooKeeper: KRaft controllers, broker roles, listeners, rack awareness, sizing and rolling upgrades, with a runnable election stand-in.
  2. 24 Monitoring, Lag and Troubleshooting Learn which Kafka metrics matter, how to read consumer lag with kafka-consumer-groups, what under-replicated partitions mean, and how to debug common failures.
  3. 25 Security and Quotas Secure Kafka with TLS encryption, SASL authentication (PLAIN, SCRAM, OAUTHBEARER), ACLs, quotas and topic naming for shared clusters, with a runnable ACL stand-in.
  4. 26 The AdminClient API and Topic Management Manage Kafka topics from code and CLI: create, describe, delete, add partitions, alter configs, consumer-group admin and the rules that make some changes irreversible.
  5. 27 Multi-Cluster Kafka: MirrorMaker 2 and Geo-Replication Copy topics between Kafka clusters with MirrorMaker 2: active-passive and active-active layouts, topic prefixes, offset translation and failover, with runnable stand-ins.
  6. 28 Testing Kafka Applications Test Kafka producers, consumers and Streams topologies: unit tests with mocks, TopologyTestDriver, Testcontainers with a real broker, and contract tests, with runnable stand-ins.
  7. 29 Running Kafka on Kubernetes Run Kafka on Kubernetes with the Strimzi operator: custom resources, StatefulSets, persistent storage, rack awareness, listeners and rolling updates, with runnable stand-ins.

Patterns

  1. 30 Event-Driven Patterns Learn event sourcing, CQRS, the outbox pattern, change data capture, sagas and event design with Kafka, using runnable outbox and event-store stand-ins.
  2. 31 Retries, Dead Letter Topics and Poison Messages Handle failing Kafka messages: retry topics with backoff, dead letter topic design, ordering trade-offs, idempotent handlers and backpressure, with runnable stand-ins.
  3. 32 Kafka with Spring Boot Use Apache Kafka from Spring Boot: KafkaTemplate, @KafkaListener, application.yml settings, NewTopic beans, retries and dead-letter topics, with a runnable stand-in.

Projects

  1. 33 Project: Order Processing Pipeline Build an order pipeline step by step: a keyed producer, a partitioned orders topic, a fulfilment consumer group with a crash, and idempotent handling, with runnable stand-ins.
  2. 34 Project: Real-Time Page-View Analytics Build a streaming analytics pipeline: page views counted per minute in tumbling windows, late events, a state store with a changelog, and an output topic, with runnable stand-ins.
  3. 35 Project: Database Sync with Change Events Turn every row change in a database into a Kafka event, keep a compacted topic as a materialized view and rebuild a lost copy by replaying it, with runnable stand-ins.
  4. 36 Project: Build a Mini Kafka Broker Build a miniature Kafka in one Node.js file: segmented partition logs, keyed produce, fetch by offset, consumer groups, generations and commits, with runnable code.

Practice and Review

  1. 37 Practice: Exercises with Solutions Eight graded Kafka exercises on lag, partitioning, consumer groups, commit order, compaction, replication and design, each with a full solution and runnable code.
  2. 38 Kafka Interview Questions Common Apache Kafka interview questions with model answers: partitions, ordering, consumer groups, delivery guarantees, replication, design and troubleshooting.
  3. 39 Kafka Cheat Sheet A compact Apache Kafka cheat sheet with CLI commands, producer, consumer, topic and broker settings, delivery recipes and decision tables for quick lookup.