Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 36 ยท Brokers and Backends

Message Brokers Compared: Redis, RabbitMQ and SQS

Compare the message brokers Celery supports, understand delivery guarantees, what a broker must do, and how to choose between Redis, RabbitMQ, SQS and others.

The broker is the most important dependency in a Celery system. Every task call passes through it, and if it is down, no new work can be queued. Choosing and configuring it well is worth an hour of thought.

What a broker does

For Celery, a broker must be able to:

  1. Accept published messages into named queues, durably enough for your needs.
  2. Hand each message to one consumer (a worker) at a time.
  3. Track whether the consumer acknowledged the message, and redeliver it if the consumer disappears.
  4. Optionally support routing, priorities and delayed delivery.

Celery talks to brokers through a library called Kombu, which hides the differences behind one interface. You choose the broker with the URL scheme:

Python
app.conf.broker_url = "redis://localhost:6379/0"            # Redis
app.conf.broker_url = "amqp://guest:guest@localhost:5672//" # RabbitMQ
app.conf.broker_url = "sqs://"                              # Amazon SQS
app.conf.broker_url = "memory://"                           # in-process, tests only

The main choices

Feature RabbitMQ Redis Amazon SQS
Protocol AMQP 0-9-1 Redis lists and sorted sets HTTPS API
Status in Celery Stable, fully featured Stable, with caveats Stable, fewer features
Durability Strong: persistent queues and confirms Depends on persistence settings Managed, very durable
Priorities Native Emulated, a few levels None
Remote control / monitoring events Full support Supported Limited
Setup effort A bit more Very easy Zero servers, AWS account
Latency Low Very low Higher (polling)
Typical use Large or critical systems Most small and mid-size apps AWS-native apps

Other transports exist (Kafka is not a real Celery broker; Azure Service Bus, Google Pub/Sub, Zookeeper and SQL databases via Kombu have varying support), but those three cover nearly every project.

Delivery guarantees in plain words

  • At-most-once: a message is delivered zero or one time. A crash can lose it. This is what early acknowledgement gives you.
  • At-least-once: delivered one or more times. A crash causes redelivery. This is what late acknowledgement gives you.
  • Exactly-once does not exist as a general guarantee over a network. You approximate it with at-least-once plus idempotent tasks, as in Task design rules.

The stand-in below implements both modes on one tiny broker interface. A worker "crashes" while processing a message; the only difference is when the ack is sent.

Python Stand-in: at-most-once vs at-least-once

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With early acknowledgement job-A is lost: the broker believed it was done. With late acknowledgement the broker still held it as unacknowledged and handed it out again. If the crash had happened after the work but before the ack, late ack would run it twice, which is why idempotency matters.

Two designs, one interface

Redis and RabbitMQ implement the same idea very differently, which explains their strengths:

flowchart LR
    subgraph Redis
      P1["producer"] -->|"LPUSH"| L["list 'celery'"]
      L -->|"BRPOP"| W1["worker"]
    end
    subgraph RabbitMQ
      P2["producer"] --> X["exchange"]
      X -->|"routing key"| Q1["queue A"]
      X --> Q2["queue B"]
      Q1 --> W2["worker"]
    end

Redis is a data store used as a queue: a list per queue. RabbitMQ is a real broker with exchanges that route messages to queues.

This runnable sketch shows the Redis style: a queue is just a list, producers push on the left and workers pop from the right (a FIFO). Celery's Redis transport emulates priorities 0 to 9 with one list per level and checks them in order, where 0 is the highest priority. The gotcha: a task sent without a priority counts as 0, so if you want "urgent" to win you must send bulk work with a larger number.

Python Stand-in: a Redis-style list queue with priority levels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The urgent tasks jump ahead of the newsletters queued first. It is coarse, though: only a handful of levels work well, and a backlog that a worker has already prefetched cannot be overtaken.

How to choose

Start with Redis if:

  • you already run Redis, or want the quickest setup;
  • tasks are short, you need decent speed and can tolerate rare edge cases;
  • you also want Redis as the result backend and cache.

Choose RabbitMQ if:

  • messages are valuable and loss is unacceptable;
  • you need real priorities, complex routing, or many different queues and consumers;
  • you want solid monitoring through events and the management UI;
  • you run long delayed tasks (hours of ETA) at scale.

Choose SQS if:

  • you run on AWS and want no servers to manage;
  • you accept polling latency and the lack of priorities and remote control;
  • you want pay-per-use and a service that scales by itself.

A good default for a new project: Redis for development and small production systems, RabbitMQ when the system becomes business critical. Because Kombu hides the transport, switching later is mostly a configuration change plus testing; the main code stays the same.

Things to configure on any broker

  • Connection retry: broker_connection_retry_on_startup=True so workers wait for the broker instead of crashing at boot.
  • Heartbeats and timeouts: let workers detect dead connections (broker_heartbeat for AMQP).
  • Pool limits: broker_pool_limit controls how many connections your producers keep. A web server with many threads can exhaust a small broker.
  • Security: use TLS (rediss:// or amqps://), strong passwords and network rules. Never expose a broker to the internet.
  • Persistence and memory: know what happens to queued messages if the broker restarts or runs out of memory.
  • Monitoring: queue length is the most important graph you can have.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What library does Celery use to talk to different brokers?

    Kombu is the messaging library that provides one interface over Redis, AMQP, SQS and others.
  2. 2. Which acknowledgement mode can lose a task if the worker crashes mid-run?

    With early ack the broker already considers the message done, so a crash during the task loses it.
  3. 3. Which broker has native, full-featured priority support?

    RabbitMQ supports priority queues natively. Redis emulates a few levels, and SQS has none.
  4. 4. When is the memory:// broker appropriate?

    It keeps messages in the process's memory, so nothing is shared and nothing survives a restart.

Summary

  • The broker stores messages, hands them to workers and tracks acknowledgements; Kombu provides a common interface.
  • RabbitMQ is the most capable choice, Redis the easiest, SQS the zero-ops AWS option.
  • Early ack gives at-most-once, late ack gives at-least-once; exactly-once is approximated with idempotent tasks.
  • Configure retry on startup, TLS, pool limits and queue-length monitoring.

Next: Using Redis as a Celery Broker