Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 39 ยท Data Formats and the Ecosystem

Kafka from Python

Use Kafka from Python with confluent-kafka: produce, delivery callbacks, flush, poll loops, group.id, auto.offset.reset, SSL/SASL settings, and kafka-python compared.

Python is common for data pipelines, services and notebooks that read from or write to Kafka. The Kafka project does not ship a Python client, so you choose a library. This lesson uses confluent-kafka, a thin wrapper over the C library librdkafka, and shows how its callback-and-poll style maps onto the concepts you already know from Writing a Producer and Writing a Consumer.

Real versus stand-in. The Python code is real usage, shown read-only because the site cannot reach a Kafka cluster (install with pip install confluent-kafka). The TryIt blocks are plain JavaScript stand-ins that imitate the shape of the client: produce that returns at once, callbacks that run during poll or flush, and a poll loop. They are not the library.

Which Python library?

Library Notes
confluent-kafka Fast (C core), maintained by Confluent, supports transactions, Schema Registry helpers, Avro/Protobuf/JSON Schema serializers and an AdminClient. The usual production choice.
kafka-python Pure Python, simple API (KafkaProducer, KafkaConsumer). Easy to read and install, slower, and with a slower release history. Fine for small tools and learning.
aiokafka asyncio client for async web apps.

The configuration names in confluent-kafka are the librdkafka names, which mostly match the Java names (bootstrap.servers, acks, group.id, enable.idempotence).

A producer

Python
from confluent_kafka import Producer
import json

producer = Producer({
    "bootstrap.servers": "localhost:9092",
    "acks": "all",
    "enable.idempotence": True,
})

def on_delivery(err, msg):
    if err is not None:
        print(f"delivery failed: {err}")
    else:
        print(f"stored in {msg.topic()}[{msg.partition()}] at offset {msg.offset()}")

for order_id in range(3):
    producer.produce(
        topic="orders",
        key=f"customer-{order_id % 2}",
        value=json.dumps({"id": order_id}).encode("utf-8"),
        on_delivery=on_delivery,
    )
    producer.poll(0)      # serve delivery callbacks that are ready

producer.flush()          # wait for everything still in the buffer

Three details matter:

  • produce() is asynchronous. It copies the record into the client's queue and returns. It raises BufferError if the local queue is full, so you may need to call poll() and retry.
  • Callbacks do not run by themselves. They run inside poll() or flush(). A producer that never calls poll accumulates reports and never tells you about failures.
  • flush() blocks until all queued records are delivered or fail. Call it before exit, just like close() in Java.

Keys and values are bytes or strings; the client does not serialize objects for you unless you use the SerializingProducer helpers with a Schema Registry (see Schemas, Avro, Protobuf and JSON).

Stand-in: produce, poll, flush

JS Delivery callbacks run during poll and flush

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A consumer

Python
from confluent_kafka import Consumer, KafkaError

consumer = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "order-service",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,
})
consumer.subscribe(["orders"])

try:
    while True:
        msg = consumer.poll(1.0)           # wait up to 1 second
        if msg is None:
            continue
        if msg.error():
            if msg.error().code() == KafkaError._PARTITION_EOF:
                continue
            raise RuntimeError(msg.error())
        print(msg.key(), msg.value().decode("utf-8"))
        consumer.commit(message=msg)       # commit after processing
finally:
    consumer.close()                       # leave the group cleanly
  • group.id is required: it names the consumer group (see Consumer Groups).
  • auto.offset.reset only applies when the group has no committed offset: earliest starts at the oldest retained record, latest (the default) at the end.
  • With enable.auto.commit off, you commit after processing, which gives at-least-once delivery (Offsets and Commits). consumer.commit(message=msg) is synchronous by default, so for throughput commit every N messages or use asynchronous=True.
  • poll() returns one message, an error message or None. Always call close() so the group rebalances at once instead of waiting for a session timeout.

Stand-in: the poll loop and commits

JS A poll loop with manual commits

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The restart re-reads the record that was handled but not committed (at-least-once), and the new group on latest sees nothing that was already in the log.

Security settings

The same client talks to a secured cluster by adding options (see Security and Quotas):

Python
conf = {
    "bootstrap.servers": "broker1.example.com:9093",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.username": "orders-app",
    "sasl.password": os.environ["KAFKA_PASSWORD"],
    "ssl.ca.location": "/etc/ssl/certs/ca.pem",
}

For mutual TLS use security.protocol: SSL with ssl.certificate.location and ssl.key.location. Read secrets from the environment or a secret manager, never from source code.

Gotchas

  • Run the consumer loop in one thread; the client object is not meant to be shared across threads for poll.
  • A long processing time between poll() calls beyond max.poll.interval.ms gets the consumer kicked out of the group.
  • Handle SIGTERM by breaking the loop and calling close().
  • kafka-python option names differ (bootstrap_servers, group_id, auto_offset_reset), so do not copy configs between libraries blindly.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When does a Producer's on_delivery callback run in confluent-kafka?

    Delivery reports are queued and served by poll() or flush(). Without calling them you never see the results.
  2. 2. What does auto.offset.reset=earliest affect?

    It only matters when there is no committed offset (or it is out of range). Otherwise the group resumes from its commit.
  3. 3. Why call consumer.close() on shutdown?

    close() leaves the group, so partitions are reassigned right away instead of after the session timeout.
  4. 4. Which library is a pure-Python client with a simpler API?

    kafka-python is pure Python. confluent-kafka wraps the C library librdkafka.

Summary

  • confluent-kafka is the usual production Python client; kafka-python is simpler and slower; aiokafka is for asyncio.
  • produce() is asynchronous; callbacks run in poll() and flush(); always flush before exiting.
  • A consumer needs group.id; auto.offset.reset applies only without a committed offset; commit after processing for at-least-once.
  • Security uses security.protocol, sasl.* and ssl.* options, with secrets from the environment.

Next: see how topics and groups are managed from code in The AdminClient API.