Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 70 ยท Serverless and messaging

SQS, SNS and Fan-Out

Learn SQS queues, visibility timeout, dead-letter queues and FIFO, plus SNS topics and the fan-out pattern, with runnable queue and topic emulators.

When one part of a system calls another directly, both must be up at the same moment, and a slow receiver slows the sender down. A message queue removes that coupling. The sender drops a message into a queue and moves on; the receiver picks messages up whenever it has capacity. If the receiver is down for an hour, messages wait. If traffic spikes, the queue absorbs it.

AWS has two core services for this. Amazon SQS (Simple Queue Service) is a queue: each message is processed by one consumer. Amazon SNS (Simple Notification Service) is a topic: each message is pushed to all subscribers. Combined, they give you the fan-out pattern. This lesson follows Lambda Functions, since Lambda is a very common consumer.

SQS and SNS cannot run on this site, and there is no AWS account here. The CLI commands are read-only. The runnable examples are small Python emulators with a fake clock that follow the same rules (visibility timeout, receive counts, dead-letter queues, FIFO groups, filter policies). They leave out durability, scale and real timing.

SQS: how a queue works

A producer sends a message with SendMessage. A consumer asks for messages with ReceiveMessage, processes them, and then calls DeleteMessage. The delete step is essential: SQS cannot know whether your processing succeeded, so it keeps every message until you tell it to remove it.

When a consumer receives a message, SQS does not remove it. It makes it invisible to other consumers for the visibility timeout (30 seconds by default, up to 12 hours). If the consumer deletes the message within that time, it is gone for good. If the consumer crashes or is too slow, the timeout ends, the message becomes visible again, and another consumer gets it.

sequenceDiagram
    participant P as Producer
    participant Q as SQS queue
    participant C as Consumer
    P->>Q: SendMessage
    C->>Q: ReceiveMessage
    Q-->>C: message (now invisible)
    C->>C: process
    C->>Q: DeleteMessage
    Note over Q: if no delete before the visibility timeout,<br/>the message becomes visible again

A message stays in the queue until it is deleted; the visibility timeout only hides it while someone works on it.

Standard queues have nearly unlimited throughput, deliver messages at least once (duplicates happen) and keep order only on a best-effort basis. Both properties push you toward the same rule you saw for Lambda: make consumers idempotent. Other defaults worth knowing: messages are kept for 4 days (configurable up to 14), and long polling (WaitTimeSeconds up to 20) lets a consumer wait for messages instead of hammering the queue with empty requests, which also lowers cost.

Dead-letter queues

Some messages can never be processed: bad data, a bug, a missing dependency. If they stay in the queue they will be retried forever and clog it. A dead-letter queue (DLQ) is a second queue that receives messages that failed too often. You attach it with a redrive policy that sets maxReceiveCount: when a message has been received that many times without being deleted, SQS moves it to the DLQ. Then an alarm on the DLQ's depth tells you something is wrong, and you can inspect the messages and replay them after fixing the bug.

The emulator below follows these rules. The consumer "crashes" on the poisoned message m2 and never deletes it.

Python Visibility timeout, retries and a dead-letter queue

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Messages m1 and m3 are deleted at once. m2 is received at t=1, stays hidden for 30 seconds and returns at t=31 and t=62. After three receives it is moved to the DLQ on the next poll, so the main queue becomes empty and healthy again.

Choose the visibility timeout a little longer than your consumer's normal processing time. If you use Lambda as the consumer, it should be at least as long as the function timeout (AWS recommends a multiple of it), otherwise a slow run will be processed twice.

FIFO queues

A FIFO queue (its name ends in .fifo) keeps messages in order and avoids duplicates:

  • Message group ID: messages in the same group are delivered in order, one at a time, so a slow message holds back only its own group. Different groups are processed in parallel.
  • Deduplication ID: a message with the same ID as one sent in the previous 5 minutes is accepted but not delivered a second time. You can also enable content-based deduplication, which hashes the body.

FIFO has lower throughput than standard (hundreds to thousands of messages a second per queue, depending on batching and mode), so use it only where order or exactly-once processing within the window really matters, such as the steps of one order or one customer's account events.

Python FIFO: message groups and deduplication

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The producer's retry of d2 is dropped. While order-1 created is being processed, the next order-1 message stays blocked, but the order-2 message is delivered in parallel. Only after the first message is deleted does order-1 paid become available. Order is therefore guaranteed within a group and free across groups.

SNS: topics and subscriptions

SNS is publish/subscribe. A publisher sends a message to a topic; SNS pushes a copy to every subscription. Subscribers can be SQS queues, Lambda functions, HTTP or HTTPS endpoints, email, SMS, mobile push notifications and Kinesis Data Firehose. Unlike a queue, SNS keeps nothing for later: a subscriber that is unreachable gets retries according to its delivery policy and then, at most, a dead-letter queue.

A filter policy on a subscription lets it receive only the messages it cares about, based on message attributes, so the publisher does not need to know who listens to what.

Fan-out: SNS plus SQS

The classic pattern puts an SQS queue in front of each consumer and subscribes all of those queues to one topic. A single publish then feeds several independent workflows, and each workflow has its own buffer, retries and DLQ.

flowchart LR
    Shop["Order service"] -->|"publish"| T["SNS topic<br/>order-events"]
    T --> Q1["SQS queue<br/>billing"]
    T --> Q2["SQS queue<br/>shipping"]
    T -->|"filter: priority = high"| Q3["SQS queue<br/>alerts"]
    Q1 --> L1["Lambda: invoices"]
    Q2 --> L2["Lambda: label printer"]
    Q3 --> L3["Lambda: notify team"]

One published event reaches every subscribed queue; a filter policy narrows what the alerts queue receives.

The beauty is decoupling. Adding a fourth consumer means adding a queue and a subscription. The order service is not changed or redeployed.

Python Fan-out with filter policies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Billing has no filter and receives everything. Shipping only wants created and paid orders, so the refund never reaches it. Alerts receives only high-priority events. Each queue can now be consumed at its own pace.

Real commands to build this are below (read-only). Note that the SQS queue also needs an access policy that lets the topic send messages to it, which aws sns subscribe does not set up for you:

Terminal
aws sns create-topic --name order-events
aws sqs create-queue --queue-name billing
aws sns subscribe \
  --topic-arn arn:aws:sns:eu-west-1:123456789012:order-events \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:eu-west-1:123456789012:billing \
  --attributes '{"FilterPolicy": "{\"priority\": [\"high\"]}", "RawMessageDelivery": "true"}'

When to use which

  • Use SQS alone to buffer work for one group of workers (image resizing, email sending) and to smooth spikes.
  • Use SNS alone to notify several systems or people immediately (email, SMS, HTTP webhooks).
  • Use SNS plus SQS when several independent consumers each need every event, reliably.
  • Need rules, schedules or events from AWS services? Look at EventBridge, covered next.
  • Very large messages do not belong in either service. Store the data in S3 and send a pointer.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A consumer receives an SQS message but crashes before deleting it. What happens next?

    SQS keeps messages until they are deleted. After the visibility timeout expires, the message is handed to a consumer again.
  2. 2. What does a dead-letter queue with maxReceiveCount = 5 do?

    The redrive policy moves repeatedly failing messages out of the main queue so they do not block it, and lets you inspect them.
  3. 3. In a FIFO queue, how is order guaranteed?

    Order is preserved within a group, and different groups can be processed in parallel.
  4. 4. What is the main benefit of the SNS plus SQS fan-out pattern?

    Each subscribed queue decouples one consumer, and the publisher does not know who listens.
  5. 5. What does an SNS filter policy do?

    A filter policy on a subscription selects which messages that subscriber receives, based on message attributes.

Summary

  • SQS decouples producers and consumers: messages stay until deleted, and the visibility timeout hides them while a consumer works.
  • Standard queues are at-least-once and loosely ordered, so make consumers idempotent; FIFO queues add per-group ordering and 5-minute deduplication at lower throughput.
  • A dead-letter queue with a redrive policy catches messages that fail too many times; alarm on its depth.
  • SNS pushes each published message to all subscribers; filter policies narrow what each one receives.
  • Fan-out (SNS topic to several SQS queues) gives every consumer its own buffer, retries and DLQ.

Next: EventBridge and Step Functions