Skip to content
Tutorials Center
My learning

Topics

Lesson 64 of 70 ยท Real-world projects

Project: Event-Driven Image Pipeline

Build an image pipeline on AWS: S3 upload events feed an SQS buffer, Lambda resizes images, DynamoDB stores metadata and a dead-letter queue catches poison files.

In this project, a user uploads a photo and a few seconds later a thumbnail and a metadata record exist, without any server waiting for the upload. S3 announces the new object, a queue absorbs the bursts, a Lambda function does the resizing, DynamoDB remembers what was produced, and a dead-letter queue collects the files that can never be processed. Every piece is small; the interesting part is how they behave when things go wrong.

The real services need an AWS account, so every command, policy and handler below is read-only. The two Python stand-ins are simplified emulators of the queue, the function and the table: they show the control flow (retries, visibility timeouts, duplicates, poison messages), not real timings. For the building blocks, see S3 Static Websites and Events, SQS, SNS and Fan-Out, Lambda Functions and DynamoDB Fundamentals.

The target architecture

flowchart LR
    U["User"] -->|"PUT uploads/cat.jpg"| S3["S3 bucket"]
    S3 -->|"ObjectCreated event"| Q["SQS queue"]
    Q --> L["Lambda: resize"]
    L -->|"write thumbs/cat.jpg"| S3
    L -->|"put item"| D["DynamoDB table"]
    Q -->|"after 3 failed receives"| DLQ["Dead-letter queue"]
    DLQ --> A["CloudWatch alarm"]

An upload triggers an event; the queue decouples the burst from the workers; failures end up in the DLQ.

Why a queue between S3 and Lambda? S3 can call Lambda directly, and for light workloads that is fine. A queue in the middle gives you three things: buffering (a thousand uploads in a second become a steady stream the function can handle), retry control (you choose the visibility timeout and how many attempts a message gets), and a dead-letter queue you can inspect and replay.

Step 1: the bucket and its rules

One bucket, two prefixes: originals go under uploads/, results under thumbs/. Keeping them separate matters, as you will see in a moment.

Terminal
# Read-only sketch: names are placeholders
aws s3api create-bucket --bucket photos-demo-1234 --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1

aws s3api put-public-access-block --bucket photos-demo-1234 \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

Nothing here is public. Users upload with presigned URLs (see S3 Buckets and Objects) and the thumbnails are served later through CloudFront or presigned downloads.

The recursion trap

If the function writes thumbs/cat.jpg into the same bucket and the bucket notifies on every new object, the thumbnail creates a new event, which creates a thumbnail of the thumbnail, forever. This is a classic way to produce a surprise bill. Two defenses, and you should use both:

  1. Configure the notification with a prefix filter (uploads/) so only originals trigger the pipeline.
  2. Have the function refuse keys outside uploads/ as a second guard.

Step 2: the queue, the DLQ and the redrive policy

Create the dead-letter queue first, then the main queue that points to it.

Terminal
# Read-only sketch
aws sqs create-queue --queue-name image-jobs-dlq \
  --attributes MessageRetentionPeriod=1209600        # keep failures for 14 days

aws sqs create-queue --queue-name image-jobs --attributes '{
  "VisibilityTimeout": "180",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-west-1:111122223333:image-jobs-dlq\",\"maxReceiveCount\":\"3\"}"
}'

Two numbers deserve care:

  • Visibility timeout must be longer than the function's timeout. If the function can run 30 seconds, a visibility timeout of 180 is comfortable (AWS suggests at least six times the function timeout when Lambda polls the queue). If it is shorter, a second worker receives the same message while the first is still working and you do the work twice.
  • maxReceiveCount says how many times a message may be received before SQS moves it to the DLQ. A file that crashes the resizer three times is almost certainly corrupt, so stop trying.

The queue also needs a queue policy that lets S3 send messages, restricted to your bucket so nobody else can inject jobs:

JSON
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "s3.amazonaws.com" },
    "Action": "sqs:SendMessage",
    "Resource": "arn:aws:sqs:eu-west-1:111122223333:image-jobs",
    "Condition": {
      "ArnLike": { "aws:SourceArn": "arn:aws:s3:::photos-demo-1234" },
      "StringEquals": { "aws:SourceAccount": "111122223333" }
    }
  }]
}

Then tell the bucket where to send events:

Terminal
# Read-only sketch
aws s3api put-bucket-notification-configuration --bucket photos-demo-1234 \
  --notification-configuration '{
    "QueueConfigurations": [{
      "QueueArn": "arn:aws:sqs:eu-west-1:111122223333:image-jobs",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": { "Key": { "FilterRules": [{ "Name": "prefix", "Value": "uploads/" }] } }
    }]
  }'

Step 3: the Lambda function

The function reads batches from SQS through an event source mapping. Each SQS message body is a JSON document that itself contains S3 Records. Three details trip people up:

  • Object keys in S3 events are URL-encoded: a space becomes +, and รฉ becomes %C3%A9. Decode with unquote_plus before using the key.
  • S3 and SQS both deliver at least once, so the same upload can arrive twice. The function must be idempotent: processing it again must not corrupt anything.
  • With batches, report failures per message (ReportBatchItemFailures) so one bad image does not send its eight healthy neighbours back to the queue.
Python
# Read-only: the shape of the real handler (Pillow comes from a layer)
import json, urllib.parse, boto3
from PIL import Image

s3 = boto3.client("s3")
table = boto3.resource("dynamodb").Table("ImageMetadata")

def handler(event, context):
    failures = []
    for message in event["Records"]:
        try:
            body = json.loads(message["body"])
            for rec in body.get("Records", []):
                bucket = rec["s3"]["bucket"]["name"]
                key = urllib.parse.unquote_plus(rec["s3"]["object"]["key"])
                if not key.startswith("uploads/"):
                    continue                      # recursion guard
                s3.download_file(bucket, key, "/tmp/in")
                with Image.open("/tmp/in") as img:
                    img.thumbnail((256, 256))
                    img.save("/tmp/out.jpg", "JPEG")
                s3.upload_file("/tmp/out.jpg", bucket, key.replace("uploads/", "thumbs/", 1))
                table.put_item(
                    Item={"imageId": key, "status": "DONE", "thumb": key.replace("uploads/", "thumbs/", 1)},
                    ConditionExpression="attribute_not_exists(imageId)",
                )
        except table.meta.client.exceptions.ConditionalCheckFailedException:
            pass                                  # duplicate delivery: already done
        except Exception:
            failures.append({"itemIdentifier": message["messageId"]})
    return {"batchItemFailures": failures}

The conditional put_item is the idempotency trick: the first delivery writes the record; a duplicate fails the condition and is treated as success. The execution role needs only s3:GetObject on uploads/*, s3:PutObject on thumbs/*, dynamodb:PutItem on that table, and the SQS read actions (ReceiveMessage, DeleteMessage, GetQueueAttributes). See IAM Best Practices and Least Privilege.

Step 4: the DynamoDB table

The access pattern is "look up an image by its id", so the key is simple:

Terminal
# Read-only sketch
aws dynamodb create-table --table-name ImageMetadata \
  --attribute-definitions AttributeName=imageId,AttributeType=S \
  --key-schema AttributeName=imageId,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

On-demand capacity suits a spiky pipeline: you pay per request and nothing when idle. If you later need "all images of user 7", you would add a sort key or an index; see DynamoDB Data Modeling.

Try it: the whole pipeline with a virtual clock

This emulator models the queue (visibility timeout and redrive to a DLQ), the function and the table. Time is a simple counter, not real seconds. The workload contains a healthy file, a corrupt file (a "poison message") and an upload that is delivered twice.

Python Queue, resizer, table and DLQ working together

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output from the top. The healthy file is stored on the first receive. The duplicate delivery of the same upload is recognized and does no harm. The corrupt file fails three times, each time reappearing only after the visibility timeout, and then lands in the DLQ. Note how the key uploads/cat photo.jpg travelled through the event as cat+photo.jpg and was decoded before use.

Operating the pipeline

A pipeline you cannot observe is a pipeline you will debug at 3 a.m. Set up these alarms in CloudWatch (see CloudWatch Metrics, Logs and Alarms):

  • ApproximateNumberOfMessagesVisible on the DLQ greater than 0: something is failing permanently.
  • ApproximateAgeOfOldestMessage on the main queue above a threshold: workers are falling behind.
  • Lambda Errors and Throttles, and DynamoDB throttled requests.

When a DLQ message appears, read it, find the cause, fix the function or delete the bad file, and then use DLQ redrive (a console feature and an API) to move the messages back to the main queue.

Also control concurrency. A burst of 5,000 uploads can start hundreds of function instances at once and overwhelm anything downstream. On the event source mapping, set maximum concurrency to cap how many instances read the queue; the queue simply absorbs the rest. This is the main reason to prefer a queue over direct S3-to-Lambda invocation for heavy work (Lambda Concurrency, Cold Starts and Limits).

Try it: choosing timeouts and caps

A small calculator shows why the numbers in Step 2 matter. Given the function's timeout, how long a batch takes and the burst size, it checks the visibility timeout and estimates how long the backlog takes to drain.

Python Check visibility timeout and estimate the drain time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The trade-off is visible: a low concurrency cap protects downstream systems but drains slowly, while an uncapped burst drains fast and can overload them. Pick the cap from what your slowest dependency can take.

Cost and cleanup

Everything here is pay-per-use, so an idle pipeline costs almost nothing, and the free tier covers a learning exercise. Two habits keep it that way: add a lifecycle rule that expires thumbs/ and uploads/ test files, and delete the stack when you are done. Put the resources in a CloudFormation or CDK stack (CloudFormation) so one command removes the queues, function, table and event wiring, and empty the bucket first because a non-empty bucket cannot be deleted.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why should the S3 notification use a prefix filter such as uploads/ when the function writes thumbnails to the same bucket?

    Without a filter, every thumbnail written to the bucket is a new ObjectCreated event, which starts the function again and again.
  2. 2. What should the visibility timeout of the queue be relative to the function timeout?

    If the message becomes visible while the first attempt is still running, another worker picks it up and the work is done twice.
  3. 3. Why is the DynamoDB write done with attribute_not_exists(imageId)?

    S3 events and SQS deliver at least once, so the function must be idempotent. A conditional put records each image exactly once.
  4. 4. What does the dead-letter queue with maxReceiveCount 3 do?

    A message that keeps failing is probably a poison message. The DLQ keeps it for inspection and replay instead of letting it loop forever.
  5. 5. Why does an S3 event key like cat+photo.jpg need decoding before you call GetObject?

    Keys in event notifications are URL-encoded. Decode with unquote_plus (or the equivalent in your language) or you will request an object that does not exist.

Summary

  • Upload events flow from S3 to an SQS queue to a Lambda function, which writes results back to S3 and metadata to DynamoDB.
  • The queue gives buffering, controlled retries and a dead-letter queue for poison messages; size the visibility timeout above the function timeout.
  • Guard against recursion with a prefix filter plus a check in the code, and decode URL-encoded keys.
  • Delivery is at least once, so make the handler idempotent, for example with a conditional put.
  • Alarm on the DLQ and on queue age, cap concurrency to protect downstream systems, and manage everything in a template so cleanup is one command.

Next: Project: Containerized App with CI/CD.