Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 39 ยท Operations

Security and Quotas

Secure Kafka with TLS encryption, SASL authentication (PLAIN, SCRAM, OAUTHBEARER), ACLs, quotas and topic naming for shared clusters, with a runnable ACL stand-in.

A freshly started Kafka broker trusts everybody. Anyone who can reach port 9092 can read every topic, write garbage into any of them and delete what they like. That is fine on a laptop and unacceptable anywhere else. Securing Kafka means answering three separate questions: can anyone listening on the network read the data (encryption), who is this client (authentication), and what may this client do (authorization). A fourth topic, quotas, protects the cluster from a client that is allowed to do something but does it far too much.

Real versus stand-in. The configuration and commands are real Kafka usage, shown read-only because the site cannot run a secured cluster. Use placeholder names and your own certificates and secrets. The TryIt blocks are stand-ins in JavaScript: an ACL evaluator and a quota throttle that follow the same rules in simplified form.

Four layers of protection

Encryption TLS hides data on the wire.
Authentication SASL or client certificates prove who you are.
Authorization ACLs say what you may do.
Quotas Limits stop one client from hogging the cluster.
1 / 2

Encryption with TLS

TLS (Transport Layer Security) encrypts traffic between clients and brokers, between brokers, and optionally between controllers. You give each broker a certificate and a private key in a keystore, and clients trust the certificate authority (CA) that signed it, through a truststore.

A listener's protocol decides what is used. The four values are PLAINTEXT (nothing), SSL (TLS only), SASL_PLAINTEXT (login without encryption, avoid it) and SASL_SSL (TLS plus login, the usual choice). They are mapped in the broker config:

PROPERTIES
listeners=EXTERNAL://:9094,INTERNAL://:9092,CONTROLLER://:9093
listener.security.protocol.map=EXTERNAL:SASL_SSL,INTERNAL:SASL_SSL,CONTROLLER:SASL_SSL
inter.broker.listener.name=INTERNAL

ssl.keystore.location=/etc/kafka/tls/broker.keystore.p12
ssl.keystore.type=PKCS12
ssl.keystore.password=${file:/etc/kafka/secrets/tls.properties:keystore.password}
ssl.truststore.location=/etc/kafka/tls/ca.truststore.p12
ssl.truststore.type=PKCS12

The ${file:...} syntax is a config provider: Kafka's way of reading a secret from a separate file at startup instead of writing it into the main config (see "Secrets" below). The matching client settings are the truststore and the protocol:

PROPERTIES
security.protocol=SASL_SSL
ssl.truststore.location=/etc/app/ca.truststore.p12
ssl.truststore.type=PKCS12

Two details trip people up. Certificates are checked against the host name the client connects to, so the certificate must list the broker's advertised name (see Running a Cluster). And TLS costs some CPU and removes some kernel-level optimizations for sending data to clients, so measure throughput on your own hardware.

Authentication

Authentication decides who a client is. The result is a principal such as User:alice, and everything afterwards (ACLs, quotas) is expressed in terms of that principal.

Mutual TLS (mTLS) has the broker ask for a client certificate (ssl.client.auth=required). The principal is derived from the certificate's distinguished name, and ssl.principal.mapping.rules can shorten it to something readable. It works well when you already operate a certificate authority.

SASL (Simple Authentication and Security Layer) is a pluggable login step. Enable the mechanisms you accept with sasl.enabled.mechanisms:

Mechanism How it works Notes
PLAIN Client sends username and password The password crosses the wire in clear text inside the exchange. Use only with TLS. Credentials usually live in broker config or a custom callback.
SCRAM-SHA-256 / SCRAM-SHA-512 Salted challenge-response; the password itself is never sent Credentials are stored (salted and hashed) by the cluster and managed with kafka-configs.sh. A solid default for application users.
GSSAPI Kerberos Common in enterprises that already run Kerberos.
OAUTHBEARER Client presents a bearer token from an OAuth 2 identity provider Tokens expire and are validated by the broker, so there are no long-lived passwords in Kafka.

A client using SCRAM:

PROPERTIES
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="order-service" \
  password="<read from a secret store>";

A SCRAM credential is created with the configuration tool, which stores a salted hash, not the password:

Terminal
bin/kafka-configs.sh --bootstrap-server broker1:9094 --command-config admin.properties \
  --alter --add-config 'SCRAM-SHA-512=[password=<new secret>]' \
  --entity-type users --entity-name order-service

Here admin.properties holds the connection settings of an administrator user.

Authorization with ACLs

Authorization is the check "may this principal do this operation on this resource?". In KRaft clusters you switch it on in the broker config:

PROPERTIES
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin
allow.everyone.if.no.acl.found=false

super.users bypass all checks, so keep that list tiny. With the authorizer on and allow.everyone.if.no.acl.found=false (the default), the rule is deny unless an ACL allows it.

An ACL has five parts: a principal (who), a host (from where, usually *), an operation (Read, Write, Create, Describe, Alter, Delete, and others), a resource (a topic, a consumer group, the cluster, a transactional id) and a permission (Allow or Deny). The resource name can be a literal name, * for all, or a prefix.

A service that writes orders, and a service that reads them in a consumer group, need these ACLs:

Terminal
# The writer
bin/kafka-acls.sh --bootstrap-server broker1:9094 --command-config admin.properties --add \
  --allow-principal User:order-service --operation Write --operation Describe --topic orders

# The reader needs the topic AND its group
bin/kafka-acls.sh --bootstrap-server broker1:9094 --command-config admin.properties --add \
  --allow-principal User:fulfilment --operation Read --operation Describe --topic orders
bin/kafka-acls.sh --bootstrap-server broker1:9094 --command-config admin.properties --add \
  --allow-principal User:fulfilment --operation Read --group fulfilment

A consumer needs Read on the group because it commits offsets there (see Offsets and Commits). Forgetting that is the classic reason a consumer can see its topic but fails to join. A transactional producer additionally needs Write on its transactional.id (see Transactions and Exactly-Once). Add --resource-pattern-type prefixed to grant a whole family of topics at once, such as everything starting with payments..

The evaluation rules, in the simplified form the stand-in follows:

  1. A super user is always allowed.
  2. If any matching Deny ACL exists, the request is denied. Deny beats allow.
  3. Otherwise, if any matching Allow ACL exists, it is allowed.
  4. Otherwise it is denied.
  5. Read, Write, Delete and Alter also imply Describe.
JS ACL evaluation

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice User:analytics: a prefixed allow covers payments.refunds, but an explicit deny on payments.cards still wins over it. Real Kafka also matches hosts and treats the wildcard principal User:* as "everybody"; the stand-in keeps only the principal wildcard.

Quotas

Quotas stop a runaway client from using the whole cluster. They are set per user, per client id, or per both, and the broker enforces them by slowing the client down: instead of rejecting requests it delays its responses (it reports the delay to the client as a throttle time). Three common limits:

  • producer_byte_rate: bytes per second a client may write, counted per broker.
  • consumer_byte_rate: bytes per second a client may fetch, per broker.
  • request_percentage: the share of broker request-handling time the client may use.
Terminal
bin/kafka-configs.sh --bootstrap-server broker1:9094 --command-config admin.properties \
  --alter --add-config 'producer_byte_rate=1048576,consumer_byte_rate=2097152' \
  --entity-type users --entity-name analytics

That gives the analytics user about 1 MiB/s of writes and 2 MiB/s of reads on each broker. The stand-in below shows the mechanism with a token-bucket-like calculation: a client that sends more than its allowance per second is told to wait.

JS A byte-rate quota

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Real brokers measure over several small windows rather than one second, but the effect is the same: the client slows down because its responses come back late, and other clients keep their share.

Topic naming in shared clusters

When many teams share one cluster, a naming convention makes authorization and ownership manageable. A common shape is <domain>.<team-or-service>.<dataset>.<version>, for example payments.checkout.order-paid.v1. Prefixed ACLs can then grant a team its whole domain, quotas can be set per team's service user, and anyone reading a topic list can see who owns what. Avoid mixing . and _ in names: Kafka metric names turn periods into underscores, so a.b and a_b can collide. Consider a rule that only automation creates topics, so naming and settings stay consistent.

Secrets

  • Never commit passwords, keystores or tokens. Inject them at deploy time from a secret store.
  • Restrict file permissions on keystores and properties files that contain secrets.
  • Use config providers or environment-based templating so the main config contains a reference, not the secret.
  • Prefer short-lived credentials (OAUTHBEARER tokens, rotated SCRAM passwords) and separate users per application so one leak has a small blast radius and an audit log is meaningful.
  • Give each application the least privilege it needs: write to its own topics, read from the ones it consumes, nothing else.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A consumer can list the topic but fails to join its group with an authorization error. Which ACL is most likely missing?

    Joining a group and committing offsets require Read on the group resource, in addition to Read on the topic.
  2. 2. An ACL allows User:analytics to read topics with the prefix payments., and another ACL denies it Read on payments.cards. What happens when it reads payments.cards?

    If any matching Deny exists, the request is denied regardless of Allow rules.
  3. 3. Why should SASL/PLAIN only be used together with TLS?

    PLAIN transmits the credentials as they are. TLS encrypts the connection so nobody on the network can read them.
  4. 4. How does a broker enforce a producer byte-rate quota?

    Quota violations are handled by throttling: the response is held back for the computed delay and the client reports it as throttle time.

Summary

  • Secure Kafka in layers: TLS for encryption, mTLS or SASL for authentication, ACLs for authorization, quotas for fairness.
  • SASL_SSL is the usual listener protocol. Prefer SCRAM or OAUTHBEARER over PLAIN, and never use PLAIN without TLS.
  • ACLs are deny-by-default; deny beats allow; consumers need Read on their group as well as the topic; prefixed patterns grant families of topics.
  • Quotas on users or client ids throttle by delaying responses.
  • Use a topic naming convention in shared clusters, and keep secrets out of config files and version control.

Next, put these building blocks to work in Event-Driven Patterns.