Skip to content
Tutorials Center
My learning

Topics

Lesson 103 of 111 ยท Projects

Project: A Message Queue with RabbitMQ and Compose

Build a producer and scalable workers around RabbitMQ with Docker Compose: health-gated startup, durable queues, acknowledgements, graceful shutdown and scaling.

A message queue sits between the part of an application that creates work and the part that does it. The producer drops a message on the queue and moves on; workers take messages one by one. If the workers are slow, messages wait. If a worker dies, someone else picks the message up. This is how email sending, image resizing and report generation are usually done.

In the Redis worker project the queue was a Redis list. Here you will use RabbitMQ, a real message broker with acknowledgements, durable queues and a management UI. You will wire it up with Docker Compose, make the workers wait until the broker is healthy, scale them to three, and watch them shut down without losing a message. Everything below was built and run on Docker 29 and Compose v5.

TEXT
 producer --publish--> +----------+ --deliver--> worker 1
 (one-off job)         | RabbitMQ | --deliver--> worker 2
                       |  "tasks" | --deliver--> worker 3
                       +----------+
                            |
                    volume "rabbitdata"

The project layout

TEXT
queue-demo/
  compose.yaml
  Dockerfile
  requirements.txt
  common.py
  producer.py
  worker.py

Step 1: Shared code

Both programs need to connect to the broker. Even with a health check, a connection can fail briefly, so a small retry loop is cheap insurance.

Python
# common.py
import os
import time

import pika

QUEUE = "tasks"


def connect(retries=10):
    params = pika.URLParameters(os.environ["AMQP_URL"])
    for attempt in range(1, retries + 1):
        try:
            return pika.BlockingConnection(params)
        except pika.exceptions.AMQPConnectionError:
            print(f"broker not ready (attempt {attempt}), retrying", flush=True)
            time.sleep(2)
    raise SystemExit("could not reach the broker")

AMQP_URL comes from the environment, so the code runs unchanged on your laptop and in production (Environment and Config). requirements.txt contains one line: pika==1.3.2, a pure-Python AMQP client.

Step 2: The producer

Python
# producer.py
import json
import sys

import pika

from common import QUEUE, connect

count = int(sys.argv[1]) if len(sys.argv) > 1 else 10
connection = connect()
channel = connection.channel()
channel.queue_declare(queue=QUEUE, durable=True)

for n in range(1, count + 1):
    channel.basic_publish(
        exchange="",
        routing_key=QUEUE,
        body=json.dumps({"id": n, "seconds": 1}),
        properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent),
    )
print(f"published {count} tasks", flush=True)
connection.close()

Two settings make the queue survive a broker restart: durable=True keeps the queue definition, and delivery_mode=Persistent writes each message to disk. Using the default exchange ("") with the queue name as routing key delivers straight to that queue.

Step 3: The worker

Python
# worker.py
import json
import signal
import socket
import time

from common import QUEUE, connect

NAME = socket.gethostname()
connection = connect()
channel = connection.channel()
channel.queue_declare(queue=QUEUE, durable=True)
channel.basic_qos(prefetch_count=1)


def handle(ch, method, properties, body):
    task = json.loads(body)
    print(f"[{NAME}] working on task {task['id']}", flush=True)
    time.sleep(task["seconds"])
    ch.basic_ack(delivery_tag=method.delivery_tag)
    print(f"[{NAME}] finished task {task['id']}", flush=True)


def stop(signum, frame):
    print(f"[{NAME}] stopping after the current task", flush=True)
    channel.stop_consuming()


signal.signal(signal.SIGTERM, stop)
channel.basic_consume(queue=QUEUE, on_message_callback=handle)
print(f"[{NAME}] waiting for tasks", flush=True)
channel.start_consuming()
connection.close()

Three details carry the design:

  • basic_ack after the work. The broker keeps a delivered message as "unacknowledged" until the worker acknowledges it. If the connection drops before the ack (the process crashes, the container is killed), RabbitMQ puts the message back on the queue for another worker. Work is never silently lost, but it can be repeated, so make tasks safe to run twice.
  • prefetch_count=1. Without it, the broker would push many messages to one worker at once, and a slow worker would hoard work while others sit idle. With 1, a worker gets a new message only after it acks the previous one.
  • The SIGTERM handler. docker stop sends SIGTERM first. The handler tells the consumer to stop after the current message and the script ends normally, so no task is cut in half (Signals and Init). The hostname in each log line is the container ID, so you can tell workers apart.

Step 4: The Dockerfile

One image serves both roles. Only the command differs.

DOCKERFILE
FROM python:3.13-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY common.py producer.py worker.py ./
USER nobody
CMD ["python", "worker.py"]

Dependencies are installed before the code is copied, so editing a Python file reuses the cached pip install layer (Build Cache). USER nobody keeps the process unprivileged (Non-root and Capabilities). If you forget a COPY, you get ModuleNotFoundError at start-up, which is exactly what happened the first time this project was built.

Step 5: The Compose file

YAML
# compose.yaml
services:
  rabbitmq:
    image: rabbitmq:4-management
    environment:
      RABBITMQ_DEFAULT_USER: app
      RABBITMQ_DEFAULT_PASS: app-secret
    ports:
      - "127.0.0.1:15672:15672"     # management UI, localhost only
    volumes:
      - rabbitdata:/var/lib/rabbitmq
    healthcheck:
      test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
      interval: 5s
      timeout: 10s
      retries: 12
      start_period: 20s

  worker:
    build: .
    image: queue-demo
    environment:
      AMQP_URL: amqp://app:app-secret@rabbitmq:5672/
    depends_on:
      rabbitmq:
        condition: service_healthy
    stop_grace_period: 15s
    restart: unless-stopped

  producer:
    build: .
    image: queue-demo
    command: ["python", "producer.py", "10"]
    environment:
      AMQP_URL: amqp://app:app-secret@rabbitmq:5672/
    depends_on:
      rabbitmq:
        condition: service_healthy
    profiles: ["tools"]

volumes:
  rabbitdata:

What to notice:

  • Health-gated start. RabbitMQ needs several seconds to boot. The healthcheck runs rabbitmq-diagnostics -q ping, and condition: service_healthy makes Compose start workers only afterward (Depends On and Health). Without it the workers would crash and restart in a loop until the broker was ready.
  • Same image: name for two services. Both build the same Dockerfile. Giving them one image name means a single build and no stale copy. (Without it Compose builds a separate image per service, and docker compose run producer can use an old one.)
  • The tools profile. The producer is a one-off job, so it is not started by docker compose up; you run it on demand (Compose Services).
  • Port bound to 127.0.0.1. The management UI at http://localhost:15672 is for you only. The broker's AMQP port 5672 is not published at all, because the workers reach it over the Compose network by service name (Container Communication).
  • Credentials in the file are for the demo. For real use, load them from an env file or secrets (Secrets Management).

Step 6: Run it and scale it

Start the broker and three workers:

Terminal
docker compose up -d --build --scale worker=3 rabbitmq worker
docker compose ps
TEXT
SERVICE    STATUS
rabbitmq   Up 2 minutes (healthy)
worker     Up 11 seconds
worker     Up 9 seconds
worker     Up 13 seconds

Publish ten tasks (each takes one second):

Terminal
docker compose run --rm producer
docker compose logs worker --no-log-prefix

The three workers share the load, and with prefetch_count=1 each one takes the next message only when it is free:

TEXT
[8bd24e2bca27] working on task 1
[8bd24e2bca27] finished task 1
[8bd24e2bca27] working on task 6
[8aedaabf7381] working on task 3
[8aedaabf7381] finished task 3
[0625e20e4213] working on task 2
[0625e20e4213] finished task 2
...

Ask the broker what it sees:

Terminal
docker compose exec rabbitmq rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers
TEXT
name    messages_ready  messages_unacknowledged consumers
tasks   0               0                       3

messages_ready are waiting, messages_unacknowledged are being worked on, and consumers is the number of connected workers. The management UI shows the same numbers with graphs.

To scale up or down, change --scale worker=N and repeat up -d. The queue has no idea how many workers there are; it just hands out messages.

Step 7: Stop gracefully, lose nothing

Publish a lot of work, then stop the workers while they are busy:

Terminal
docker compose run --rm producer python producer.py 90
docker compose stop -t 15 worker
docker compose exec rabbitmq rabbitmqctl list_queues name messages_ready messages_unacknowledged consumers
TEXT
name    messages_ready  messages_unacknowledged consumers
tasks   18              0                       0

Each worker printed stopping after the current task and exited with code 0. Nothing was in flight (messages_unacknowledged is 0), and 18 of the 90 messages are still waiting. Start the workers again and they drain the rest:

Terminal
docker compose up -d --scale worker=3 worker

If a worker is killed instead of stopped (docker kill, out of memory, a host crash), the broker notices that the connection closed and requeues whatever it had not acknowledged. That is the safety net behind acknowledgements, and the reason tasks should be idempotent.

Finally, clean up, including the volume:

Terminal
docker compose --profile tools down -v

A Redis variant

You can swap the broker for Redis when you do not need acknowledgements, routing or a management UI. The producer does RPUSH tasks <json>, and the worker does BLPOP tasks 0 in a loop (the Redis worker project). Compared with RabbitMQ:

Redis list RabbitMQ
Setup One tiny container A heavier broker with a health check
Lost work if a worker crashes mid-task Yes, the message was already removed (unless you use BLMOVE and a second list) No, unacked messages are requeued
Routing (topics, fan-out) Build it yourself Exchanges and bindings
Persistence Optional, configurable Durable queues and persistent messages
Good for Simple background jobs, caches that double as queues Reliable job delivery, several consumers and routes

Both fit into the same Compose shape: broker, workers that depend on it, and a one-off producer.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the worker call basic_ack only after finishing the task?

    An unacknowledged message stays with the broker. If the connection closes before the ack, RabbitMQ delivers it to another worker, so the work is not lost.
  2. 2. What does condition: service_healthy in depends_on do?

    Compose waits for the dependency's healthcheck to report healthy before it starts the dependent service, so workers do not crash while the broker boots.
  3. 3. What is the effect of channel.basic_qos(prefetch_count=1)?

    With prefetch 1 a worker receives a new message only after it acknowledges the previous one, so fast workers take more tasks and slow workers are not overloaded.
  4. 4. What happens to messages in a durable queue when RabbitMQ restarts, if the volume is kept?

    Durable queues and persistent messages are written to disk in /var/lib/rabbitmq. Because that directory is a named volume, the data survives container recreation.

Summary

  • A queue separates producing work from doing it; Compose runs the broker, scalable workers and a one-off producer from one image.
  • A health check plus depends_on: condition: service_healthy orders start-up; a small retry loop covers the rest.
  • Durable queues, persistent messages and a volume keep messages across restarts; acknowledgements after the work keep them across crashes.
  • prefetch_count=1 spreads work fairly; a SIGTERM handler lets workers finish the current task on docker compose stop.
  • Scale with --scale worker=N. Make tasks idempotent, because a message can be delivered more than once.
  • Redis lists are the lighter option when you can accept losing in-flight work.

Next: Project: Multiple Apps Behind a Proxy.