Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 39 ยท Patterns

Retries, Dead Letter Topics and Poison Messages

Handle failing Kafka messages: retry topics with backoff, dead letter topic design, ordering trade-offs, idempotent handlers and backpressure, with runnable stand-ins.

Sooner or later a consumer meets a record it cannot process. Sometimes the cause is temporary: the database is restarting, a downstream API timed out. Sometimes it is permanent: the JSON is malformed, a required field is missing, a business rule can never be satisfied. Kafka itself does not decide what happens next. A consumer reads records in order from its offset, and if the code simply throws and retries the same record forever, the partition is stuck behind it. That record is called a poison message.

This lesson shows the options: retry in place, retry through separate topics with a delay, and park hopeless records in a dead letter topic (DLT, also called a dead letter queue or DLQ). It also covers what each choice does to ordering, and how to avoid overwhelming a struggling system.

Real versus stand-in. The ideas and the consumer-API calls named here (pause, resume, seek) are real Kafka usage. The TryIt blocks are JavaScript stand-ins with a virtual clock instead of real consumers and topics, so the outputs are deterministic. Libraries such as Spring Kafka offer retry-topic and dead-letter support ready-made; check your client library before writing your own.

A record fails. Now what?

Retry in place Try again right away a few times.
Retry topics Park it, wait, try again later.
Dead letter topic Give up and keep it for humans.
Skip Only if losing it is acceptable.
1 / 2

Classify the failure

Everything starts with one question: could this succeed if I tried again later?

  • Transient failures (timeouts, connection resets, "database is locked", rate limits) are worth retrying, with a pause.
  • Permanent failures (a deserialization error, a missing mandatory field, a value that violates a business rule) will fail identically every time. Retrying wastes time and, worse, delays everything behind them. Send these straight to the dead letter topic.

Your handler should turn exceptions into one of those two categories explicitly, for example by throwing a RetriableError or a PermanentError. Unknown exceptions are usually treated as transient a limited number of times and then dead-lettered, so a bug cannot loop forever.

Option 1: retry in place

The simplest approach: when the handler fails, wait a little and call it again for the same record, up to a limit, before moving on. It preserves order, since nothing else is processed meanwhile, and needs no extra topics. The cost is that the partition makes no progress while you wait, and consumers must keep calling poll often enough to stay in the group: if processing plus waiting exceeds max.poll.interval.ms, the consumer is removed and a rebalance starts (see Consumer Groups and Rebalancing). Keep in-place retries short, a few attempts over a few seconds.

Option 2: retry topics with backoff

For failures that need more time to heal, move the record out of the way so the main topic keeps flowing. You create a chain of topics, each representing a wait:

orders โ†’ orders-retry-1s โ†’ orders-retry-5s โ†’ orders-dlt

When the handler fails on the main topic, the consumer publishes the record to the first retry topic (with headers saying how many attempts it has had and when it may next run) and then commits its offset on the main topic. A separate consumer reads the retry topic, and if the record is not yet due, it does not process it: it waits (by pausing the partition with pause() and resuming later with resume(), so it stays in the group). On success the record is done; on failure it moves to the next, longer topic; after the last one it goes to the dead letter topic.

Kafka has no built-in delayed delivery, so the delay is always implemented by the consumer. Fixed tiers of a few topics keep the design simple; a delay that doubles each time is called exponential backoff, and adding a random jitter stops many failing records from retrying at the same instant.

Retry topics and a dead letter topic, simulated

The stand-in uses a virtual clock in milliseconds. Four records arrive: one fine, one that fails twice and then succeeds (a transient problem), one malformed (permanent), and one transient failure that never heals. Watch each take its path.

JS Retry chain and DLT

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that the permanent failure reached the DLT at once, while the stubborn transient one used up every tier first. The healthy record was never delayed by the others.

Designing the dead letter topic

A DLT is not a trash can; it is an inbox for humans and tools. A few rules make it useful:

  • Keep the original record unchanged (key, value, headers) and add headers that explain the failure: the original topic, partition and offset, the exception message, the number of attempts and a timestamp. Without them nobody can tell why it is there.
  • Alert on it. A DLT that grows silently is data loss with extra steps. Track its size and its arrival rate.
  • Plan for reprocessing. After you fix the bug or the data, you need a way to send records back to the main topic (a small tool or job). That is only safe if handlers are idempotent.
  • Decide its retention. Records that nobody looks at for a week may as well be deleted; records that must never be lost need a longer retention than the topic they came from.
  • Use one DLT per consumer group or per topic, and say which in the name (orders.fulfilment.dlt). A shared DLT mixes unrelated failures.
  • Mind privacy. The DLT holds a copy of the original payload, so it needs the same access controls (see Security and Quotas).

Ordering trade-offs

Kafka orders records only within a partition. The moment you move a failed record to a retry topic, the main consumer carries on and handles later records for the same key before the failed one comes back. That breaks per-key order.

Whether this matters depends on your data. For independent records (payments for different customers, page views) it is harmless. For a stream of updates to the same entity (OrderPlaced, then OrderCancelled) it can be dangerous: the retried OrderPlaced could be applied after the cancellation. Your options, from strictest to most relaxed:

  1. Block the partition (retry in place), keeping order and accepting stalls.
  2. Retry topics plus a guard: once a key has a record waiting for retry, send all later records of that key to the retry path as well, so order within the key is kept. This needs a small lookup of keys in retry.
  3. Retry topics and a version check: let records run out of order, but make the handler ignore anything older than the state it already holds, using a version number or event timestamp.
  4. Relax the need for order: design events so they can be applied in any order.

The second stand-in shows option 3. Two updates for the same order arrive; the first fails once and is retried later, after the second update has already been applied.

JS Out-of-order retries and a version guard

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Without the guard the order ends up "placed" even though it was cancelled. With it, the stale retry is recognized and dropped.

Idempotent handlers

Retries mean the same record can be handled more than once, and a crash between "side effect done" and "offset committed" has the same effect (see Offsets and Commits). So handlers must be idempotent: running them twice has the same result as once. Common techniques:

  • Use natural keys and upserts: INSERT ... ON CONFLICT DO UPDATE rather than a blind insert.
  • Store the processed event id (or topic-partition-offset) in the same transaction as the business change, and skip ids already seen.
  • Pass an idempotency key to external APIs that support one, such as payment providers.
  • Prefer "set status to shipped" over "increment counter".

Backpressure

When a downstream system is slow or failing, the worst reaction is to keep hammering it. Kafka's pull model helps: a consumer only receives what it asks for. Use that deliberately.

  • Lower max.poll.records so one batch does not take longer than max.poll.interval.ms.
  • Use pause(partitions) when your internal work queue is full or a circuit breaker is open, keep calling poll (it returns nothing for paused partitions, and the consumer stays in the group), and resume(partitions) when there is room again.
  • Cap concurrency to the downstream system, and apply backoff and jitter to retries, so a recovering service is not flattened by a retry storm.
  • Watch lag (see Monitoring, Lag and Troubleshooting): growing lag with a busy dependency is the signal to slow down, scale out or shed load, not to retry faster.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A record has an invalid payload that can never be parsed. What should the consumer do?

    A permanent failure will not heal. Retrying only blocks the partition, so park the record with its error details and move on.
  2. 2. Why can retry topics break ordering for a key?

    The failed record leaves the main flow, so newer records for the same key can overtake it. Use a guard such as a version check, or keep that key on the retry path.
  3. 3. What is the main purpose of jitter in a backoff strategy?

    Randomizing the delay spreads retries out, so a recovering service is not hit by a synchronized wave.
  4. 4. Which handler is idempotent?

    An upsert keyed by a natural id gives the same state no matter how often it runs. The others change the outcome on every duplicate.
  5. 5. How does a consumer slow down a busy downstream system without leaving its group?

    A paused partition returns no records, but continued poll calls keep the consumer alive in the group, avoiding a rebalance.

Summary

  • Classify failures: transient ones deserve a delayed retry, permanent ones go straight to a dead letter topic.
  • Retry in place keeps order but stalls the partition; retry topics keep the main flow moving at the price of ordering.
  • A good DLT keeps the original record, adds failure headers, raises alerts and has a reprocessing path.
  • Protect per-key order with guards (version checks) or by routing a key's later records through the retry path.
  • Make handlers idempotent and apply backoff with jitter; use pause/resume and smaller batches for backpressure.

Next, combine everything in Project: Order Processing Pipeline.