Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Projects and Practice

Project: Order Processing Pipeline

Design and build an event-driven order pipeline with RabbitMQ: topic exchange, per-service quorum queues, retries, a dead letter queue, publisher confirms, and a runnable mini-broker simulation.

In this project you design a small but realistic order pipeline: a shop publishes order.placed, and three services react: billing charges the customer, warehouse reserves stock, notifications sends the confirmation email. Failures must not lose orders, and a payment that keeps failing must end up somewhere a human can see it. You will use what the previous lessons taught: exchanges and routing, confirms, acks, durability, dead lettering and quorum queues.

RabbitMQ is not available on this page. The real topology and pika code are shown read-only, and the runnable blocks use a plain Python mini-broker written for this lesson: it implements exchanges, bindings, acks and dead lettering in about 60 lines. It is a teaching model, not RabbitMQ, but the design decisions you make are the same.

Requirements

  1. The shop must never lose an order: publish with persistence, confirms and mandatory.
  2. Each service gets every order and processes it independently; one service failing must not affect the others.
  3. A transient failure (payment gateway timeout) is retried after a delay; a permanent one (invalid card) goes straight to a dead letter queue.
  4. Operations can see what failed and replay it.

The topology

flowchart LR
    Shop[Shop API] -->|order.placed| X((orders<br/>topic))
    X -->|order.placed| QB[billing.orders<br/>quorum]
    X -->|order.placed| QW[warehouse.orders<br/>quorum]
    X -->|order.#| QN[notifications.orders<br/>quorum]
    QB -->|reject| DLX((orders.dlx))
    QW -->|reject| DLX
    QN -->|reject| DLX
    DLX --> DLQ[orders.failed<br/>quorum]
    QB --> B[Billing workers]
    QW --> W[Warehouse workers]
    QN --> N[Notification workers]

Declarations, run once by a setup script or each service at start-up (all idempotent):

Python
channel.exchange_declare("orders", "topic", durable=True)
channel.exchange_declare("orders.dlx", "fanout", durable=True)
channel.queue_declare("orders.failed", durable=True, arguments={"x-queue-type": "quorum"})
channel.queue_bind("orders.failed", "orders.dlx")

for service, key in [("billing", "order.placed"), ("warehouse", "order.placed"), ("notifications", "order.#")]:
    name = f"{service}.orders"
    channel.queue_declare(name, durable=True, arguments={
        "x-queue-type": "quorum",
        "x-dead-letter-exchange": "orders.dlx",
        "x-delivery-limit": 5,
    })
    channel.queue_bind(name, "orders", routing_key=key)

Notice x-delivery-limit: 5 as a safety net for poison messages, and that every service owns its queue.

Step 1: the publisher

The shop API publishes inside the request, after the order row is stored. A production version would use the outbox pattern from the pub/sub lesson, so the database commit and the event cannot diverge.

Python
channel.confirm_delivery()
channel.basic_publish(
    exchange="orders",
    routing_key="order.placed",
    body=json.dumps(order).encode(),
    properties=pika.BasicProperties(
        delivery_mode=pika.DeliveryMode.Persistent,
        content_type="application/json",
        message_id=f"order-{order['id']}",
        type="order.placed",
    ),
    mandatory=True,
)

If UnroutableError or NackError is raised, the order stays in the outbox for a later retry.

Step 2: a mini-broker to try the design

The block below contains the whole pipeline: a tiny broker (topic matching, per-queue delivery, ack/nack with requeue, dead lettering after a delivery limit) and the three consumers. One order has an invalid card (permanent failure), one hits a gateway timeout twice (transient failure), and the others succeed.

Python Stand-in: the order pipeline end to end

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Study the output. Order 2 is retried twice and then succeeds. Order 3 is dead-lettered by billing, but warehouse still reserved its stock: the services are independent, so you now have inconsistent state. That is the next design problem.

Step 3: handle partial failure with compensation

In an event-driven system there is no global transaction. When billing fails permanently, billing publishes payment.failed; warehouse subscribes to it and releases the reservation (a compensating action); the order service marks the order cancelled and notifications emails the customer. This chain of local steps and compensations is a saga. Each step must be idempotent, because events can repeat.

Python Stand-in: compensation with payment.failed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 4: operations

  • Dead letter queue: orders.failed holds permanently failed deliveries with the x-death header (original queue and reason). Alert when it is non-empty and give support a small tool to inspect and replay a message to its original queue after the bug or data is fixed.
  • Delayed retry: replace the immediate requeue for transient errors with the TTL retry queue from Dead Letter Exchanges and Retries, so a gateway outage does not create a hot loop.
  • Scaling: run several billing workers with prefetch_count around 10, and watch backlog and consumer count (Monitoring and Prometheus).
  • Security: one user per service with permissions limited to its own queue and write to nothing but what it needs (Security: Users, Virtual Hosts and TLS).
  • Failure drills: kill a node during a load test; stop the billing consumers; fill the DLQ; verify alerts fire and orders are not lost.

Extensions to try

  1. Add an analytics service reading the same events from a stream and replay a day of orders.
  2. Add order.cancelled and make warehouse release stock when it arrives.
  3. Give premium customers a priority lane: a second routing key order.placed.priority and dedicated workers.
  4. Write a contract for the event payload (fields, versions) and make consumers ignore unknown fields.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does each service have its own queue bound to the orders exchange?

    An exchange copies events into every bound queue; separate queues isolate services from each other.
  2. 2. A payment fails permanently after stock was already reserved. What is the right fix?

    Without distributed transactions you compensate each step with another event (a saga).
  3. 3. What is the purpose of x-delivery-limit on the service queues?

    After the limit, the quorum queue dead-letters the message to the DLX instead of redelivering it forever.
  4. 4. Why must compensating event handlers be idempotent?

    At-least-once delivery means duplicates; applying a release twice would corrupt stock.

Summary

  • A reliable pipeline combines persistent, confirmed publishing with per-service quorum queues, manual acks and a dead letter queue.
  • Separate transient failures (retry with delay) from permanent ones (dead-letter immediately).
  • Independent services produce partial failures; solve them with compensating events and idempotent handlers.
  • Operate it: alert on the DLQ, watch backlog, practice failures.

Next: Project: Task Queue with Retries and Idempotency