Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Practice: Troubleshooting Stuck Queues

Practice diagnosing RabbitMQ problems from symptoms: growing ready or unacked counts, missing consumers, unroutable messages, blocked connections, redelivery loops and PRECONDITION_FAILED.

Real RabbitMQ incidents rarely announce their cause. You get a symptom, "emails are late", "the queue keeps growing", "the publisher hangs", and have to work backward. This practice lesson gives you a diagnostic method, a catalog of the usual suspects with the numbers that betray them, and exercises with sample data to rehearse the reasoning. The tools come from Management UI, HTTP API and CLI Tools and Monitoring and Prometheus.

RabbitMQ cannot run on this page, so shell commands are read-only, and all runnable blocks are plain Python stand-ins working on sample data.

A method that works

  1. Define the symptom with numbers. Which queue? Ready, unacked, consumers, publish rate, ack rate, message age.
  2. Locate the stage. Publisher to exchange? Exchange to queue? Queue to consumer? Consumer to ack?
  3. Check the broker's own health. Alarms, node status, partitions, disk, memory, file descriptors.
  4. Form one hypothesis, test it, change one thing.
  5. Write down the fix and add the missing alert.
Text
rabbitmq-diagnostics check_running
rabbitmq-diagnostics check_local_alarms
rabbitmqctl list_queues name type messages_ready messages_unacknowledged consumers
rabbitmqctl list_consumers
rabbitmqctl list_connections name state channels
rabbitmqctl cluster_status

The symptom table

Symptom Likely cause How to confirm Fix
Ready grows, consumers = 0 Consumers crashed, not deployed, wrong queue name or vhost list_consumers, deployment status, logs Restart or fix the consumer; check vhost and name
Ready grows, consumers > 0, ack rate low Consumers too slow CPU/IO of workers, handler timing Scale out, raise prefetch for fast jobs, speed up the handler
Unacked large and flat, ack rate 0 Stuck handlers or forgotten basic_ack Thread dump, code review Fix the ack path; add timeouts
Unacked large, prefetch huge Unlimited prefetch hoards messages prefetch in the consumers list Set basic_qos
Messages "disappear" Unroutable (no binding), drop-head overflow, TTL expiry Publish with mandatory, check alternate exchange, policies Fix bindings, use confirms/mandatory, review limits
Same message keeps returning Requeue loop, poison message redelivered=true, x-delivery-count requeue=false plus DLX; set a delivery limit
Channel closed PRECONDITION_FAILED ... delivery acknowledgement timed out Handler holds a message longer than consumer_timeout Broker log Ack earlier; shorter jobs; raise timeout
Declaring a queue fails PRECONDITION_FAILED inequivalent arg Existing queue has different arguments list_queues arguments Use the same arguments, or delete and recreate, or use a policy
Publisher hangs or connection.blocked Memory or disk alarm, or flow control check_local_alarms, UI state blocked / flow Drain, add capacity, see Flow Control and Resource Alarms
Connections keep dropping Missed heartbeats (long sync handler), load balancer idle timeout, network Broker logs, handler duration Heartbeat 30 to 60 s, work in threads, raise LB timeout
ACCESS_REFUSED Wrong user, vhost or permission regex list_permissions -p vhost Fix permissions (Security)
Queue unavailable after a node loss Classic queue on the lost node, or quorum queue without majority list_queues name leader members online Restore the node; migrate to quorum queues (Quorum Queues)
Messages out of order Several consumers, requeues, multiple publishers Message ids and timestamps Single active consumer, or design for unordered

Exercise 1: read the table of numbers

Given a snapshot of queues, name the likely problem for each.

Python Stand-in: classify queues by symptom

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise 2: where did the message go?

When a message "vanishes", walk the path: exchange exists? binding matches? queue accepts? consumer acks? The stand-in below traces a message and reports the first failing stage.

Python Stand-in: trace a vanished message

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exercise 3: the redelivery loop

A consumer logs the same order every second. Use the redelivered flag and counts to confirm a loop and decide the action.

Python Stand-in: detect a requeue loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Debugging tools and habits

  • Management UI "Get messages" to peek at a body (with requeue) and see headers such as x-death; do this on a copy or low-traffic queue.
  • rabbitmq-diagnostics for read-only truth: alarms, listeners, memory, cluster state.
  • Broker logs for PRECONDITION_FAILED, ACCESS_REFUSED, connection closures and Raft messages.
  • Client logs: log the delivery tag, redelivered flag, message id and handling duration for every message at debug level.
  • Firehose tracer (rabbitmqctl trace_on): publishes copies of all traffic to an exchange; expensive, use briefly in non-production.
  • Reproduce in a clean vhost with a script before changing production policies.

Never "fix" a stuck queue by purging it before copying the messages somewhere safe; a purge is irreversible.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Ready messages grow while consumers = 0 and publish rate is steady. What is the first thing to check?

    No consumers means nobody drains the queue: crashed or undeployed workers, or a wrong queue or vhost.
  2. 2. Unacked is high and flat, the ack rate is zero. What is the likely cause?

    Messages are delivered but never completed, so they stay unacked.
  3. 3. Messages published to an exchange never appear in any queue and there is no error. What do you check?

    Unroutable messages are dropped silently unless published with mandatory or an alternate exchange exists.
  4. 4. Why is purging a queue a poor first reaction to a stuck queue?

    Diagnose first; copy messages if needed. A purge cannot be undone.

Summary

  • Diagnose in stages: publisher, exchange, binding, queue, consumer, ack; confirm with numbers.
  • Ready growth points to missing or slow consumers; flat unacked with zero acks points to stuck handlers or forgotten acks.
  • Silent loss comes from unroutable messages, drop-head overflow and TTLs.
  • Add the alert that would have caught it, then move on.

Next: Practice: Design Review and Patterns