Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Patterns: RPC, Work Queues

Work Queues and Fair Dispatch

Build a RabbitMQ work queue with competing consumers: round-robin dispatch, fair dispatch with prefetch 1, acks, scaling workers, single active consumer, and a runnable simulation.

The work queue (also called the competing consumers pattern) is the bread and butter of RabbitMQ. A producer puts tasks on a queue, such as "resize this image", "send this email" or "generate this invoice", and a pool of workers takes them one by one. Slow jobs no longer block web requests, and when the backlog grows you simply start more workers. The queue is the buffer that smooths bursts.

RabbitMQ is not available on this page, so the pika code is read-only and the runnable blocks are plain Python stand-ins that model how the broker hands out messages. They make the dispatch rules visible; they are not RabbitMQ.

The shape of a work queue

flowchart LR
    P1[Web app] --> Q[(tasks queue)]
    P2[Scheduler] --> Q
    Q --> W1[Worker 1]
    Q --> W2[Worker 2]
    Q --> W3[Worker 3]

Each message goes to exactly one worker. Producers do not know or care how many workers exist. The setup needs four ingredients you already know: a durable queue (or a quorum queue), persistent messages, manual acks and a prefetch limit.

Producer:

Python
import json
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
channel = connection.channel()
channel.queue_declare(queue="thumbnails", durable=True)
channel.confirm_delivery()

def enqueue(job: dict) -> None:
    channel.basic_publish(
        exchange="",
        routing_key="thumbnails",
        body=json.dumps(job).encode(),
        properties=pika.BasicProperties(
            delivery_mode=pika.DeliveryMode.Persistent,
            content_type="application/json",
            message_id=job["id"],
        ),
        mandatory=True,
    )

enqueue({"id": "img-1001", "path": "uploads/a.png", "width": 320})

Worker:

Python
def on_job(ch, method, properties, body):
    job = json.loads(body)
    make_thumbnail(job["path"], job["width"])        # the slow part
    ch.basic_ack(delivery_tag=method.delivery_tag)   # only after success

channel.basic_qos(prefetch_count=1)
channel.basic_consume(queue="thumbnails", on_message_callback=on_job)
channel.start_consuming()

The producer uses the default exchange (exchange=""), which routes a message straight to the queue named by the routing key. See Queues and Exchanges in this track's earlier chapters for the exchange types.

Round-robin dispatch and why it can be unfair

With no prefetch limit RabbitMQ hands messages to consumers in turn: message 1 to worker A, 2 to B, 3 to A, and so on. That is "round-robin", and it distributes message counts evenly. If every job takes the same time, that is perfect. If jobs vary (every second job is huge), worker A could receive all the big jobs while B sits idle, because RabbitMQ does not look at how busy each worker is: it pushes a message as soon as it has one.

Python Stand-in: round-robin with uneven jobs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Worker A is busy for 40 seconds, B for 4. The pool takes as long as its slowest member.

Fair dispatch with prefetch

Setting basic_qos(prefetch_count=1) tells the broker: do not give this worker a new message until it has acknowledged the previous one. A free worker pulls the next job, so work flows to whoever is idle.

Python Stand-in: fair dispatch with prefetch 1

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With prefetch=1 the finishing time falls close to the average, because cheap jobs flow to the worker that finished its big one.

For jobs that take a long time, use prefetch_count=1. For very short jobs, raise it (10 to 50) so workers are never starved waiting for the next network round trip. Measure with your real handler.

Scaling and failure

  • Scale out by starting more worker processes. Each connects, calls basic_consume, and the broker includes it in the rotation immediately.
  • Scale in safely by letting workers finish: stop consuming (basic_cancel), finish the current job, ack, then close. Killing a worker mid-job requeues that message, which is fine because it was not acked.
  • Worker crash: the broker requeues unacked messages (see Consumer Acknowledgements); another worker picks them up. Jobs must be idempotent, because a worker may have completed half the work.
  • Autoscaling signal: the number of ready messages (backlog) and consumer count are the metrics to watch; see Monitoring and Prometheus.

Single active consumer

Sometimes you want a queue with several consumers for failover but processing in order, for instance a ledger writer. Declare the queue with x-single-active-consumer: true. Only one consumer receives messages; the others wait idle as hot standbys and one takes over when the active one disconnects. This keeps ordering across the pool at the cost of parallelism. Streams and super streams use the same idea per partition.

Priorities and other options

  • Priority queues: declare x-max-priority (classic queues, keep it between 1 and 5) and set a priority property on messages. Higher priority messages are delivered first when there is a backlog. Quorum queues support just two levels.
  • Delayed jobs: use a TTL plus dead lettering (see Dead Letter Exchanges and Retries) or the delayed-message-exchange plugin.
  • Job idempotency key: put a unique message_id on every job and record finished ids in the database to skip duplicates.
Python Stand-in: idempotent worker with message ids

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How are messages distributed to workers on one queue by default?

    Each message goes to exactly one consumer, in turn, without looking at how busy each worker is. Prefetch makes it adapt.
  2. 2. What does basic_qos(prefetch_count=1) achieve in a work queue?

    It limits unacknowledged messages per consumer to one, so idle workers pull the next job.
  3. 3. When should you use x-single-active-consumer?

    Only one consumer is active at a time, others stand by, preserving order.
  4. 4. Why should work queue jobs be idempotent?

    At-least-once delivery means duplicates are possible; an idempotent job does no harm when repeated.

Summary

  • A work queue distributes each task to exactly one of several competing workers.
  • Use durable or quorum queues, persistent messages, manual acks and publisher confirms.
  • Default round-robin ignores job size; prefetch_count=1 gives fair dispatch for long jobs.
  • Scale by adding workers, keep jobs idempotent, and use single active consumer when order matters.

Next: RPC with Reply Queues