Apache Kafka Tutorial
Learn Apache Kafka step by step: 39 lessons in 11 chapters with examples, quizzes, diagrams and projects.
Start learning →
of lessons completed
Getting Started
- What Is Apache Kafka? Learn what Apache Kafka is, how brokers, topics, producers and consumers fit together, where Kafka shines, and when a plain queue or database is better.
- Kafka vs RabbitMQ, ActiveMQ, Pulsar and Cloud Queues Compare Kafka with RabbitMQ, ActiveMQ, Apache Pulsar, Amazon SQS, Google Pub/Sub and Azure Service Bus: queues versus logs, push versus pull, replay, routing and when to choose each.
- The Commit Log Idea Understand the append-only commit log behind Kafka: offsets, immutability, replay, retention and why sequential disk writes make Kafka fast.
- Your First Topic with the CLI Start a single-node KRaft Kafka broker, create a topic with kafka-topics, and send and read messages with the console producer and consumer.
Topics and Partitions
- Topics and Partitions Learn how Kafka splits a topic into partitions, why ordering holds only inside one partition, how partition count limits parallelism, and what adding partitions does.
- Message Keys and Partitioners See how Kafka picks a partition from a record key with murmur2, how the sticky partitioner handles keyless records, and how hot keys and custom partitioners work.
- Replication, Leaders and ISR Understand Kafka replication: leaders and followers, the in-sync replica set, acks and min.insync.replicas, the high watermark, and what unclean election risks.
Producers
- Writing a Producer Write a Kafka producer in Java and JavaScript: records, serializers, send(), callbacks, acks, retriable errors and delivery timeouts, with a runnable producer stand-in.
- Batching, Compression and Idempotence Tune Kafka producer throughput with linger.ms, batch.size and compression, then prevent duplicates with enable.idempotence and sequence numbers. Runnable stand-ins included.
Consumers
- Reading with a Consumer Learn the Kafka consumer poll loop: subscribe vs assign, deserializers, auto.offset.reset, max.poll.records and graceful shutdown, with a runnable poll-loop stand-in.
- Consumer Groups and Rebalancing Understand Kafka consumer groups: partition assignment, range vs round-robin vs cooperative-sticky, heartbeats, session timeout, static membership and scaling limits.
- Offsets and Commits Learn how Kafka consumers track progress: committed offsets, __consumer_offsets, auto vs manual and sync vs async commits, seek and replay, and consumer lag, with runnable stand-ins.
Delivery Guarantees
- At-Most-Once, At-Least-Once, Exactly-Once Where Kafka duplicates and data loss come from, how commit ordering gives at-most-once or at-least-once, and how idempotent consumers and dedup keys approach exactly-once.
- Transactions and Exactly-Once Understand Kafka transactions: transactional.id, read_committed, consume-transform-produce, zombie fencing and where exactly-once stops, with runnable stand-ins.
Storage: Retention and Compaction
- Retention, Segments and Disk Layout See how Kafka stores partitions as segment files with indexes, how time and size retention delete old data, and what tiered storage changes, with runnable stand-ins.
- Log Compaction and Tombstones Learn how Kafka log compaction keeps the latest value per key, how tombstones delete keys, and how compacted topics act as tables, with a runnable stand-in.
Data Formats and the Ecosystem
- Schemas, Avro, Protobuf and JSON Why Kafka topics need schemas, how a schema registry works, what backward, forward and full compatibility mean, and how to evolve a schema safely, with runnable stand-ins.
- Kafka Connect Learn Kafka Connect: source and sink connectors, workers, tasks, converters, single message transforms, offsets and the REST API, with a runnable pipeline stand-in.
- Kafka Streams and Stream Processing Learn Kafka Streams and stream processing: KStream vs KTable, stateless and stateful operations, windowing, joins and state stores with changelog topics, with runnable stand-ins.
- Kafka from Python Use Kafka from Python with confluent-kafka: produce, delivery callbacks, flush, poll loops, group.id, auto.offset.reset, SSL/SASL settings, and kafka-python compared.
- ksqlDB: SQL over Kafka Process Kafka topics with SQL in ksqlDB: streams versus tables, persistent, push and pull queries, windowed aggregates and joins, with runnable stand-ins.
- Kafka with Spark Structured Streaming and Flink Read and write Kafka topics from Spark Structured Streaming and Flink: source and sink options, offsets and checkpoints, exactly-once sinks, with runnable stand-ins.
Operations
- Running a Cluster: KRaft, Brokers and Controllers Understand how a Kafka cluster runs without ZooKeeper: KRaft controllers, broker roles, listeners, rack awareness, sizing and rolling upgrades, with a runnable election stand-in.
- Monitoring, Lag and Troubleshooting Learn which Kafka metrics matter, how to read consumer lag with kafka-consumer-groups, what under-replicated partitions mean, and how to debug common failures.
- Security and Quotas Secure Kafka with TLS encryption, SASL authentication (PLAIN, SCRAM, OAUTHBEARER), ACLs, quotas and topic naming for shared clusters, with a runnable ACL stand-in.
- The AdminClient API and Topic Management Manage Kafka topics from code and CLI: create, describe, delete, add partitions, alter configs, consumer-group admin and the rules that make some changes irreversible.
- Multi-Cluster Kafka: MirrorMaker 2 and Geo-Replication Copy topics between Kafka clusters with MirrorMaker 2: active-passive and active-active layouts, topic prefixes, offset translation and failover, with runnable stand-ins.
- Testing Kafka Applications Test Kafka producers, consumers and Streams topologies: unit tests with mocks, TopologyTestDriver, Testcontainers with a real broker, and contract tests, with runnable stand-ins.
- Running Kafka on Kubernetes Run Kafka on Kubernetes with the Strimzi operator: custom resources, StatefulSets, persistent storage, rack awareness, listeners and rolling updates, with runnable stand-ins.
Patterns
- Event-Driven Patterns Learn event sourcing, CQRS, the outbox pattern, change data capture, sagas and event design with Kafka, using runnable outbox and event-store stand-ins.
- Retries, Dead Letter Topics and Poison Messages Handle failing Kafka messages: retry topics with backoff, dead letter topic design, ordering trade-offs, idempotent handlers and backpressure, with runnable stand-ins.
- Kafka with Spring Boot Use Apache Kafka from Spring Boot: KafkaTemplate, @KafkaListener, application.yml settings, NewTopic beans, retries and dead-letter topics, with a runnable stand-in.
Projects
- Project: Order Processing Pipeline Build an order pipeline step by step: a keyed producer, a partitioned orders topic, a fulfilment consumer group with a crash, and idempotent handling, with runnable stand-ins.
- Project: Real-Time Page-View Analytics Build a streaming analytics pipeline: page views counted per minute in tumbling windows, late events, a state store with a changelog, and an output topic, with runnable stand-ins.
- Project: Database Sync with Change Events Turn every row change in a database into a Kafka event, keep a compacted topic as a materialized view and rebuild a lost copy by replaying it, with runnable stand-ins.
- Project: Build a Mini Kafka Broker Build a miniature Kafka in one Node.js file: segmented partition logs, keyed produce, fetch by offset, consumer groups, generations and commits, with runnable code.
Practice and Review
- Practice: Exercises with Solutions Eight graded Kafka exercises on lag, partitioning, consumer groups, commit order, compaction, replication and design, each with a full solution and runnable code.
- Kafka Interview Questions Common Apache Kafka interview questions with model answers: partitions, ordering, consumer groups, delivery guarantees, replication, design and troubleshooting.
- Kafka Cheat Sheet A compact Apache Kafka cheat sheet with CLI commands, producer, consumer, topic and broker settings, delivery recipes and decision tables for quick lookup.