Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 65 ยท Asynchronous Systems

Message Queues and Asynchronous Processing

Learn message queues for system design: producers, consumers, acknowledgements, at-least-once delivery, idempotency, backpressure and scaling workers, with Python demos.

Most requests in a web system do not need to finish all their work before the user gets a response. When someone uploads a video, you do not make them wait for transcoding. When they place an order, the confirmation email can go out a moment later. Asynchronous processing means accepting the request, recording the work to be done, replying quickly and doing the slow part in the background. The tool that makes this reliable is the message queue.

A queue sits between the part of the system that creates work (the producer) and the part that does it (the consumer or worker). This lesson covers how queues work, what guarantees they give, and the mistakes that make them lose or duplicate work.

Why put a queue in the middle

What a queue buys you

Fast responses Reply now, work later.
Load leveling A spike waits in line instead of crashing you.
Decoupling Producers do not know the consumers.
Retry Failed work stays queued and can run again.

The queue absorbs differences in speed and uptime.

1 / 2
  • Load leveling: a sudden flood of 10,000 requests becomes 10,000 queued messages that workers drain at their own pace.
  • Decoupling: the checkout service publishes "order placed" without knowing about email, invoices or analytics.
  • Resilience: if the email worker is down for ten minutes, messages wait and are delivered when it returns.
  • Independent scaling: add workers when the queue grows; remove them when it empties.

The price is that the system becomes eventually complete: the user may see "processing" for a while, and you must handle duplicates and failures explicitly.

How a queue works

flowchart LR
  P1[Producer] --> Q[(Queue)]
  P2[Producer] --> Q
  Q --> W1[Worker 1]
  Q --> W2[Worker 2]
  Q --> W3[Worker 3]

Several producers add messages; competing workers each take different messages.

Messages are usually delivered first-in, first-out. Each message goes to one worker (the competing consumers pattern), which is how you share work. The message-lifecycle is:

  1. The producer enqueues a message.
  2. A worker receives it. The broker hides it for a while (the visibility timeout) rather than deleting it.
  3. The worker finishes and sends an acknowledgement (ack). Only now is the message deleted.
  4. If the worker crashes or the timeout expires with no ack, the message reappears and another worker takes it.
Python A queue with acks and visibility timeouts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Delivery guarantees

Because networks and workers fail, a broker cannot give every guarantee for free. Choose the one your use case tolerates:

Guarantee Meaning Risk
At-most-once Delivered 0 or 1 times; no retry Lost messages
At-least-once Delivered 1 or more times; retried until acked Duplicates
Exactly-once Processed exactly once Hard, costly, usually "effectively once"

Most systems use at-least-once delivery and make the consumer idempotent: processing the same message twice has the same effect as processing it once. True exactly-once delivery across a network is not achievable in general; what you can get is exactly-once effect through idempotency.

Python Duplicates are harmless when consumers are idempotent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The usual implementation stores a unique message or request ID (or uses a database uniqueness constraint) and skips IDs it has already handled.

Backpressure, ordering and poison messages

  • Backpressure: if producers outrun consumers, the queue grows without bound. Monitor queue depth and age of the oldest message; scale workers, slow producers, or reject new work when it gets too long.
  • Ordering: a single queue with one consumer is ordered. With many competing workers, messages are processed in parallel and can finish out of order. If order matters for one entity (all events for order #7), route by key so they always go to the same partition or worker.
  • Poison messages: a message that always fails would be retried forever and block progress. Limit retries, then move it to a dead letter queue for inspection (see Retries and Dead Letter Queues).
  • Visibility timeout tuning: shorter than the real processing time causes duplicate processing; too long delays recovery from crashes. Let long jobs extend the timeout with heartbeats.

Queues versus other tools

A queue is for work that one consumer should do. If many different services want to react to the same event, use publish/subscribe (next lesson). If you need to replay history, use a log-based system, covered in Event Sourcing and Streaming. Common implementations include RabbitMQ, Amazon SQS, Redis lists and streams, and Kafka.

Return a job ID from your API and give clients a status endpoint (GET /jobs/42) or a webhook. Asynchronous only feels good when users can see progress.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A worker crashes before acknowledging a message. What does a typical queue do?

    Messages are only removed after an ack. Without one, the visibility timeout expires and the message is redelivered.
  2. 2. Why should consumers be idempotent in an at-least-once system?

    Redelivery after timeouts or retries creates duplicates, so handling a message twice must have the same effect as once.
  3. 3. Queue depth keeps growing. What is the first thing to check or do?

    Growth means consumers are slower than producers. Scale workers or apply backpressure.
  4. 4. What happens to a poison message without retry limits?

    A message that always fails keeps coming back. Retry limits and a dead letter queue contain it.
  5. 5. Which guarantee do most message systems provide, with idempotent consumers on top?

    At-least-once with idempotent handling gives exactly-once effect, which is practical where true exactly-once delivery is not.

Summary

  • Queues decouple producers from consumers, level load and make background work reliable.
  • Workers take a message, process it and ack; no ack means redelivery after the visibility timeout.
  • At-least-once delivery plus idempotent consumers is the standard answer to duplicates.
  • Watch queue depth and message age; use backpressure, retry limits and dead letter queues.
  • Use a queue for one consumer per task; use pub/sub when many services react to one event.

Next: Publish/Subscribe and Message Brokers