Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 39 ยท Operations

Monitoring, Lag and Troubleshooting

Learn which Kafka metrics matter, how to read consumer lag with kafka-consumer-groups, what under-replicated partitions mean, and how to debug common failures.

A Kafka cluster that nobody watches fails in slow, quiet ways: a consumer falls an hour behind, a broker's disk fills, a topic loses a replica and nobody notices until the next failure makes it a real outage. Monitoring is how you see these problems while they are still cheap to fix.

You cannot watch everything, and you should not try. This lesson picks the few signals that answer the questions that matter: is data flowing, is it safe, and is anybody falling behind? Then it walks through the tools and the most common failures.

Real versus stand-in. The commands, metric names and sample output are real Kafka usage, shown read-only because the site cannot reach a cluster. The TryIt blocks are JavaScript stand-ins that do the same arithmetic the tools do (lag per partition, time to catch up) on made-up numbers.

Three questions to answer

Is it healthy? Brokers up, one active controller, no offline partitions.
Is it safe? Replicas in sync, disks not full.
Is anyone behind? Consumer lag and request latency.
1 / 2

Broker metrics that matter

Kafka brokers publish hundreds of metrics through JMX (Java Management Extensions). Monitoring stacks scrape them with an exporter or agent. These are the ones to alert on first:

Metric (JMX name, shortened) What it tells you Typical alert
UnderReplicatedPartitions Partitions where a follower is behind or missing greater than 0 for several minutes
OfflinePartitionsCount Partitions with no leader at all (controller metric) greater than 0, immediately
ActiveControllerCount Exactly one node should report 1 cluster total other than 1
UnderMinIsrPartitionCount Partitions below min.insync.replicas, so acks=all writes fail greater than 0
RequestHandlerAvgIdlePercent How busy the request threads are falling toward 0 means saturation
BytesInPerSec / BytesOutPerSec Traffic, for capacity trends compare with baseline
RequestQueueSize Requests waiting for a thread growing steadily
TotalTimeMs per request type End-to-end latency for produce and fetch p99 above your objective

The first two connect directly to Replication, Leaders and ISR. An under-replicated partition (URP) has a follower that dropped out of the in-sync set. One URP during a rolling restart is normal and clears itself. A URP that stays for ten minutes means a slow disk, a broken network path, an overloaded broker or a dead broker. An offline partition means producers and consumers of that partition are failing right now.

Also watch the machine: disk usage per log directory, disk latency, network throughput and, for the Java process, garbage collection pauses and heap use. Kafka relies heavily on the operating system's page cache, so free memory that looks "unused" is doing useful work.

Producer and consumer metrics

Clients publish their own metrics, and they are often the first to show trouble.

For producers: record-error-rate and record-retry-rate (failed or repeated sends), request-latency-avg, batch-size-avg and compression-rate-avg (are your batching settings working? see Batching, Compression and Idempotence), and buffer-available-bytes (if it hits zero, send starts blocking).

For consumers: records-lag-max (the worst lag among the partitions this client reads), fetch-latency-avg, commit-latency-avg, rebalance-total and last-poll-seconds-ago. A consumer that has not called poll for longer than max.poll.interval.ms is removed from its group, which triggers a rebalance (see Consumer Groups and Rebalancing).

Consumer lag

Lag is the most useful number in day-to-day Kafka operation. For one partition it is

lag = log end offset - committed offset

The log end offset is where the next record will be written; the committed offset is the position the group has saved (see Offsets and Commits). Lag counts records, not seconds. A lag of 10,000 is nothing for a topic receiving 50,000 records per second and a disaster for one receiving ten per hour. What you really want to know is time to catch up, which needs the rates of production and consumption.

Look at lag with the command-line tool:

Terminal
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group fulfilment
TEXT
GROUP       TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST        CLIENT-ID
fulfilment  orders  0          1500            1500            0    worker-a-1   /10.0.0.5   worker-a
fulfilment  orders  1          1210            1900            690  worker-b-1   /10.0.0.6   worker-b
fulfilment  orders  2          980             1000            20   worker-b-1   /10.0.0.6   worker-b

CURRENT-OFFSET is the committed offset. Here partition 1 is the problem: it has most of the lag while partition 0 is fully caught up. That pattern points away from "the whole group is slow" and toward a cause specific to one partition or one consumer: a hot key, a slow worker, or a stuck record.

Other useful forms of the tool are --list (all groups), --describe --state (group state and coordinator), --describe --members --verbose (who owns which partitions), and --reset-offsets (move a group's position; stop the group first and use --dry-run before --execute).

A lag calculator

This stand-in reproduces the table: given log end offsets and committed offsets, it prints lag per partition and the total, then flags the worst partition.

JS Lag per partition

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Lag over time

A single reading cannot tell a healthy group from a dying one. The trend can. If lag stays flat or shrinks, the consumers keep up. If it grows, they consume more slowly than producers write, and no restart will fix that without more capacity.

The second stand-in simulates six one-minute samples for two groups and estimates how long each needs to catch up.

JS Is the lag growing?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Billing is recovering from a backlog and will finish; analytics is losing ground even though its lag looks smaller. Alert on a lag that grows for a sustained period, or on lag measured as time (some monitoring tools estimate "how old is the oldest unconsumed record"), rather than on one absolute number.

Common failures and what to check

Symptom Likely causes What to check
Producers time out, NOT_ENOUGH_REPLICAS errors Fewer in-sync replicas than min.insync.replicas UnderMinIsrPartitionCount, broker logs, disk and network
UnderReplicatedPartitions stays above 0 Slow or dead follower, saturated disk or network Which broker is missing from the ISR (kafka-topics.sh --describe --under-replicated-partitions)
Consumer lag grows on all partitions Consumers too slow or too few Processing time per record, group size versus partition count
Lag grows on one partition Hot key, stuck record, one slow consumer Partition skew, the consumer that owns it, a poison message
Constant rebalances Slow processing, max.poll.interval.ms too low, unstable network, rolling deploys Rebalance metrics, poll gaps, static membership
Client connects, then hangs Wrong advertised.listeners Listener config (see Running a Cluster)
Duplicates after restarts At-least-once processing Commit order, idempotent handler (see At-Most-Once, At-Least-Once, Exactly-Once)
Broker out of disk Retention too long, traffic grew, compaction behind Retention settings and segment sizes (see Retention, Segments and Disk Layout)
RecordTooLargeException Record above max.request.size or the topic's max.message.bytes Reduce the payload, or store large blobs elsewhere and send a reference

Useful commands while investigating:

Terminal
# Where are the leaders and replicas of a topic?
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

# Only the troubled partitions
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-replicated-partitions
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --unavailable-partitions

# Look at the newest end offsets of a topic
bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders

# Inspect the metadata quorum (KRaft)
bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status

When a single record is the problem, your code needs a policy for it. That is the subject of Retries, Dead Letter Topics and Poison Messages.

Alerting without noise

Page a human for symptoms that hurt users or will soon: offline partitions, under-min-ISR partitions, no active controller, consumer lag growing beyond your agreed delay, disk predicted to fill within hours. Send everything else (CPU a bit high, a single URP during a deploy) to a dashboard or a ticket queue. Alerts that fire daily teach people to ignore them.

Add a small "canary": a producer and consumer that exchange a heartbeat record every few seconds and measure the round trip. It tests the whole path from the client's point of view, including things no single metric shows.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How is consumer lag for one partition computed?

    Lag is how many records exist beyond the group's committed position: log end offset minus committed offset.
  2. 2. UnderReplicatedPartitions has been 4 for twenty minutes, and nobody is restarting brokers. What does that most likely mean?

    A persistent URP means follower replicas are not keeping up or are missing. It is a durability risk that needs investigation.
  3. 3. Lag is large on partition 1 only; partitions 0 and 2 are at zero. What should you suspect first?

    If the others keep up, the group has enough capacity overall. Look at the owner of partition 1 and at the records it is reading.
  4. 4. Why is a lag value alone a weak alert?

    10,000 records may be a second of backlog or a week of it. Alert on growth over time or on estimated delay.

Summary

  • Watch three things: health (controller, offline partitions), safety (URP, under-min-ISR, disks) and progress (consumer lag, latency).
  • A persistent under-replicated partition is a durability warning; an offline partition is an outage for that partition.
  • Lag is log end offset minus committed offset; read it with kafka-consumer-groups.sh --describe, and judge it by trend and time to catch up.
  • The pattern of lag across partitions points to the cause: all partitions means capacity, one partition means skew or a stuck record.
  • Alert on symptoms, keep the rest on dashboards, and consider a canary.

Next, lock the cluster down with Security and Quotas.