Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 36 ยท Brokers and Backends

Using RabbitMQ: Exchanges, Queues and Routing Keys

Set up RabbitMQ as a Celery broker and understand AMQP exchanges, queues, bindings and routing keys, durability, priorities, virtual hosts and the management UI.

RabbitMQ implements the AMQP protocol and is the broker Celery was originally designed around. It is a dedicated message server with real routing, per-message acknowledgements and persistent queues, so it handles long delays and large queues without the tricks Redis needs.

Installing and connecting

Run it with Docker (the management image includes a web UI):

Terminal
docker run -d --name rabbit -p 5672:5672 -p 15672:15672 rabbitmq:3-management

The default user is guest with password guest, which RabbitMQ only allows from localhost. The management interface is at http://localhost:15672. Connect Celery:

Python
app.conf.broker_url = "amqp://guest:guest@localhost:5672//"

The URL is amqp://user:password@host:port/vhost. The path after the port is the virtual host; the default vhost is called /, which is why the URL ends with // (a slash separator plus the vhost name /). Use amqps:// for TLS.

In production, create a dedicated user and vhost per application:

Terminal
rabbitmqctl add_vhost myapp
rabbitmqctl add_user myapp_user 'a-strong-password'
rabbitmqctl set_permissions -p myapp myapp_user ".*" ".*" ".*"
Python
app.conf.broker_url = "amqp://myapp_user:[email protected]:5672/myapp"

A vhost is a namespace: queues and exchanges in one vhost are invisible in another, so several apps can share one RabbitMQ safely.

The AMQP model

Redis has one concept, the list. AMQP has three:

  • Exchange: where producers publish. It does not store messages; it routes them.
  • Queue: a buffer that stores messages for consumers.
  • Binding: a rule linking an exchange to a queue, usually with a routing key pattern.
flowchart LR
    P["Producer<br/>routing key: video.convert"] --> X(("exchange<br/>'tasks'"))
    X -->|"video.*"| Q1["queue: video"]
    X -->|"email.#"| Q2["queue: email"]
    X -->|"#"| Q3["queue: audit"]
    Q1 --> W1["worker A"]
    Q2 --> W2["worker B"]
    Q3 --> W3["worker C"]

A message is routed to every queue whose binding pattern matches its routing key.

There are several exchange types:

Type Routing rule
direct Exact match of routing key and binding key.
topic Pattern match on dotted keys: * matches one word, # matches zero or more words.
fanout Copies to every bound queue, ignoring the key.
headers Match on message headers (rarely used with Celery).

Celery's default is a direct exchange named celery, a queue named celery and the routing key celery: simple enough that you do not notice any of it. You only meet exchanges when you define your own queues (see Queues and routing).

See routing work

This stand-in implements the matching rules of direct, topic and fanout exchanges, so you can try patterns. The topic matcher follows AMQP semantics.

Python Stand-in: direct, topic and fanout exchanges

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the results: video.convert.hd does not match video.* (the star means exactly one word) but does match the catch-all #. A message that matches no binding is silently dropped by RabbitMQ unless you configure a mandatory flag or an alternate exchange, which is a classic source of "my task vanished".

Durability and persistence

For messages to survive a broker restart, three things must be true:

  1. The queue is durable (declared with durable=True),
  2. The exchange is durable, and
  3. The message is persistent (delivery mode 2).

Celery's default queues and messages satisfy all three, so a restart of RabbitMQ keeps queued tasks. You can see the defaults in your own definitions:

Python
from kombu import Exchange, Queue

app.conf.task_queues = (
    Queue("default", Exchange("default"), routing_key="default", durable=True),
    Queue("video", Exchange("video"), routing_key="video", durable=True),
)
app.conf.task_default_queue = "default"

If you need a faster broker and can lose messages, task_publish_retry and transient messages exist, but persistent is the sane default.

Priorities

RabbitMQ supports real priority queues. Declare the queue with a max priority and send a priority per task:

Python
Queue("default", Exchange("default"), routing_key="default",
      queue_arguments={"x-max-priority": 10})

add.apply_async((1, 2), priority=9)    # on RabbitMQ, higher number = more important

Careful: on RabbitMQ a higher number means higher priority, while Celery's Redis emulation treats 0 as highest. When you switch brokers, check your priority numbers. Keep the maximum small (about 5 to 10); each level costs resources inside RabbitMQ.

Quorum queues and high availability

Classic mirrored queues are deprecated. For HA, use quorum queues, which replicate across nodes with Raft:

Python
app.conf.task_default_queue_type = "quorum"        # Celery 5.5+
app.conf.broker_transport_options = {"confirm_publish": True}

With a three-node cluster, the queue survives the loss of one node. Quorum queues do not support all features (priorities are limited, for example), so test your setup. Also enable confirm_publish so apply_async returns only after the broker has accepted the message.

Connection health and flow control

  • Heartbeats (broker_heartbeat = 10) detect dead TCP connections so unacknowledged messages come back quickly.
  • RabbitMQ applies flow control when memory or disk is nearly full: publishers are blocked until space is free. Set broker_connection_timeout and watch alarms in the UI.
  • Long tasks and heartbeats: with the prefork pool, the worker's main process sends heartbeats, so a long task in a child does not break the connection. With the solo pool, a long task can block heartbeats; the broker then closes the connection.
  • consumer_timeout: RabbitMQ (3.8.15+) closes the channel of a consumer that holds an unacknowledged message longer than 30 minutes (default 1800000 ms). This hits acks_late tasks and long ETA tasks. Raise it in rabbitmq.conf (consumer_timeout = 7200000) if tasks legitimately run longer.

Operating it

The management UI shows, per queue: ready messages, unacknowledged messages, publish and deliver rates and consumers. From the command line:

Terminal
rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers
rabbitmqctl list_exchanges
rabbitmqctl list_bindings

If messages_ready grows while consumers is 0, nobody is listening to that queue. If messages_unacknowledged is high and stuck, workers hold messages (check prefetch and long tasks).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In AMQP, where do producers publish messages?

    Producers publish to exchanges. Bindings decide which queues receive the message.
  2. 2. What does the topic binding key email.# match?

    matches zero or more words, so email, email.welcome and email.welcome.eu all match.

  3. 3. What happens to a message whose routing key matches no binding (with default settings)?

    Unroutable messages are discarded unless you use mandatory publishing or an alternate exchange.
  4. 4. What is a virtual host (vhost)?

    Vhosts separate applications on one broker. The URL path selects the vhost.
  5. 5. Which settings make a message survive a RabbitMQ restart?

    All three must hold. Celery's defaults provide them.

Summary

  • RabbitMQ routes messages through exchanges to queues using bindings; Celery's default is a direct exchange and a queue named celery.
  • Use topic exchanges for pattern routing, fanout for broadcast; unroutable messages are dropped.
  • Create a vhost and user per app; use quorum queues for high availability and confirm publishing.
  • Watch consumer_timeout, heartbeats, and the queue counters ready and unacknowledged.

Next: Result Backends: Redis, Database, RPC and More