Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 57 ยท Real-World Projects

Project: Event-Driven Image Processing

Build an event-driven pipeline on Azure: Blob upload, Event Grid, a Storage queue and a Functions worker that makes thumbnails, with retries and poison messages.

Many useful systems follow one pattern: something happens (a file arrives, an order is placed), and the system reacts without anyone polling for it. In this project a user uploads a photo to Blob Storage and, a moment later, a thumbnail and a metadata record exist. Nobody calls the thumbnail code. The upload raises an event, the event is queued, and a worker processes the queue.

This design is small, but it teaches the habits that make cloud systems robust: decoupling the producer from the consumer, absorbing bursts in a queue, assuming every message may arrive more than once, retrying temporary failures, and setting aside messages that can never succeed. The real services need an Azure account, so their commands and code are read-only. The runnable part is a Node.js stand-in that models the queue, the retries and the poison handling with a simulated clock. You will use ideas from Storage Accounts and Blob Storage, Azure Files, Queues and Tables, Event Grid and Event Hubs and Azure Functions.

The architecture

flowchart LR
    U["User upload"] --> B1[("Blob container<br/>uploads")]
    B1 -->|"BlobCreated event"| EG["Event Grid<br/>system topic"]
    EG -->|"filter: uploads/ only"| Q["Storage queue<br/>image-events"]
    Q --> F["Function worker<br/>(queue trigger)"]
    F -->|"reads original"| B1
    F -->|"writes thumbnail"| B2[("Blob container<br/>thumbnails")]
    F -->|"writes record"| T[("Table<br/>image metadata")]
    Q -.->|"after 5 failed tries"| P["Poison queue<br/>image-events-poison"]

Two containers, one queue and one poison queue. The worker writes to a different container than the one that raises events.

Why this shape?

  • Event Grid delivers an event when a blob is created, with no polling. A system topic on the storage account exposes the built-in events such as Microsoft.Storage.BlobCreated.
  • A queue between the event and the worker gives you buffering (a thousand uploads at once do not overwhelm the worker), retries and a place to inspect failures.
  • The worker is a function with a queue trigger, so Azure scales it with the backlog (see Scaling and Autoscale).
  • Thumbnails go to a different container. If the worker wrote back into uploads, each thumbnail would raise another event and the pipeline would feed on itself forever. Filtering on the container prefix is your second defense.

There is a simpler variant: a blob trigger on the function, which watches a container directly. The classic blob trigger polls the storage account and can be slow to notice new blobs on a busy account. In the Node.js programming model you can set the blob trigger's source to Event Grid so it reacts to events instead of scanning. We build the explicit queue version because it shows each moving part and gives you the retry and poison behavior for free.

Setting up the plumbing

These Azure CLI commands are read-only here (they create billable resources):

Terminal
az group create --name rg-images --location westeurope

az storage account create --name stimages001 --resource-group rg-images \
  --sku Standard_LRS --allow-blob-public-access false

az storage container create --account-name stimages001 --name uploads --auth-mode login
az storage container create --account-name stimages001 --name thumbnails --auth-mode login
az storage queue create --account-name stimages001 --name image-events --auth-mode login

# System topic on the storage account, then a subscription that writes into the queue
az eventgrid system-topic create --name st-images --resource-group rg-images \
  --location westeurope --topic-type Microsoft.Storage.StorageAccounts \
  --source $(az storage account show -n stimages001 -g rg-images --query id -o tsv)

az eventgrid system-topic event-subscription create \
  --name to-image-queue --system-topic-name st-images --resource-group rg-images \
  --endpoint-type storagequeue \
  --endpoint $(az storage account show -n stimages001 -g rg-images --query id -o tsv)/queueservices/default/queues/image-events \
  --included-event-types Microsoft.Storage.BlobCreated \
  --subject-begins-with /blobServices/default/containers/uploads/

The subscription's filter is what keeps unrelated blobs out of the queue. The subject of an event looks like /blobServices/default/containers/uploads/blobs/holiday.jpg, so subject-begins-with selects one container. Event Grid also needs permission to write into the queue, which you set up with its identity and a queue sender role; and the function app's managed identity needs data roles on the account, such as Storage Blob Data Contributor and a queue role (see Managed Identities and Service Principals and check the Functions documentation for the exact roles your trigger type needs). Do not put account keys in settings.

Messages written by other services may not use the encoding the Functions queue trigger expects by default, so match the messageEncoding setting in host.json to what is actually in the queue.

The worker function

The worker receives the event, downloads the original, makes a thumbnail, uploads it, and records the metadata. This code is read-only (it needs the Functions runtime, the Azure SDK packages and the sharp image library):

JavaScript
const { app } = require("@azure/functions");
const { BlobServiceClient } = require("@azure/storage-blob");
const { TableClient } = require("@azure/data-tables");
const { DefaultAzureCredential } = require("@azure/identity");
const sharp = require("sharp");

const credential = new DefaultAzureCredential();
const blobs = new BlobServiceClient(process.env.BLOB_URL, credential);
const table = new TableClient(process.env.TABLE_URL, "imagemetadata", credential);

app.storageQueue("makeThumbnail", {
  queueName: "image-events",
  connection: "StorageConn",
  handler: async (event, context) => {
    // subject: /blobServices/default/containers/uploads/blobs/<name>
    const name = decodeURIComponent(event.subject.split("/blobs/")[1]);

    const source = blobs.getContainerClient("uploads").getBlobClient(name);
    const props = await source.getProperties();

    // Idempotency: skip if this exact version was already processed
    try {
      const done = await table.getEntity("images", name);
      if (done.etag === props.etag) {
        context.log(`skip ${name}: already processed`);
        return;
      }
    } catch (e) {
      if (e.statusCode !== 404) throw e;
    }

    const original = await source.downloadToBuffer();
    const image = sharp(original);
    const meta = await image.metadata();
    const thumb = await image.resize(128, 128, { fit: "inside" }).toBuffer();

    await blobs.getContainerClient("thumbnails").getBlockBlobClient(name)
      .uploadData(thumb, { blobHTTPHeaders: { blobContentType: props.contentType } });

    await table.upsertEntity({
      partitionKey: "images", rowKey: name,
      width: meta.width, height: meta.height, etag: props.etag,
    });
  },
});

Three details deserve attention:

  1. Deterministic output. The thumbnail has the same name as the original, in another container. Running the function twice for the same blob overwrites the thumbnail with identical content. Work that is safe to repeat is called idempotent.
  2. The duplicate check. Event Grid and Storage queues both promise at-least-once delivery, not exactly-once. A message can be delivered twice (for example when a worker takes longer than the visibility timeout). The metadata record, keyed by blob name and version, lets the worker skip finished work.
  3. Errors are thrown, not swallowed. If the handler throws, the message stays in the queue and reappears after the visibility timeout, for another attempt. Catching every exception and returning normally would delete the message and silently lose the image.

Retries and poison messages

The Functions queue trigger retries for you. When a function fails, the message becomes visible again after a configurable delay, up to a maximum number of attempts. In host.json:

JSON
{
  "version": "2.0",
  "extensions": {
    "queues": {
      "maxDequeueCount": 5,
      "visibilityTimeout": "00:00:30",
      "batchSize": 8
    }
  }
}

After maxDequeueCount failed attempts (5 by default), the runtime moves the message into a poison queue named after the original with a -poison suffix (image-events-poison). That is the dead-letter mechanism for Storage queues: a place where messages that cannot be processed wait for a person or a repair job. Without it, one corrupt file would be retried forever and block throughput.

Not every failure is the same:

  • Transient failures (a network blip, a throttled storage call) usually succeed on retry. Retrying is the right answer.
  • Permanent failures (a file that is not an image) will never succeed. Retrying five times wastes time, but sending the message to the poison queue and raising an alert is correct.

You cannot always tell them apart in advance, which is why a bounded number of attempts plus a poison queue is the standard design. Alert on the length of the poison queue (see Azure Monitor, Log Analytics and Application Insights).

A runnable stand-in: events, queue, worker, poison queue

The program below is not Azure. It simulates the pipeline with a virtual clock: an Event Grid topic with a subject filter, a queue with visibility timeouts and dequeue counts, a worker that "makes thumbnails" from small made-up image records, and a poison queue. The scripted scenario covers a normal upload, a duplicate event, a flaky storage call that fails twice, a corrupt file, and a blob in the wrong container.

Node.js Event pipeline with retries, duplicates and poison messages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Study the log. The duplicate event for holiday.jpg is detected and skipped, because the catalog already holds that version. party.png fails twice with a transient error, then succeeds on the third attempt after the visibility timeout expires twice. broken.jpg fails every time, and on the fifth attempt the message moves to the poison queue, where an operator can inspect it. The thumbnail written into the thumbnails container produced an event that was filtered out, so the pipeline did not loop.

Testing the thumbnail rule

The resize rule is pure logic, so test it without any cloud. This second stand-in checks the "fit inside 128 by 128" math, including the edge cases that cause bugs: very wide images, tiny images that should not be enlarged, and exact squares.

Node.js Test the thumbnail size rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Notice the panorama: without the Math.max(1, ...) guard its height would round to zero, and a zero-size image is a failure waiting to happen. The corrupt case throws, which in the real pipeline leads to retries and finally the poison queue.

Operating the pipeline

  • Monitoring. Track queue length (the backlog), the age of the oldest message, function failures and the poison queue length. A growing backlog means the worker is too slow or broken.
  • Replaying poison messages. After fixing a bug, move messages from image-events-poison back to image-events. Because the worker is idempotent, replaying is safe.
  • Security. Disable public blob access, use managed identities, and validate uploads (type, size) before processing. Image libraries parse untrusted input, so keep them updated and limit memory. Do not trust the file extension.
  • Cost. Queues and Event Grid cost fractions of a cent per thousand operations, and the Consumption plan bills only for execution time (see Pricing, Cost Management and Budgets). Add a lifecycle rule to delete old originals if you do not need them.
  • Scale and limits. Storage queue messages are limited in size, so put the blob's name in the message and never the image itself.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the worker write thumbnails to a different container than the one that raises events?

    Each created blob raises a BlobCreated event. Separate containers and a subject filter keep the output from feeding the input.
  2. 2. A Storage queue message is delivered, and the worker takes longer than the visibility timeout. What can happen?

    Queues provide at-least-once delivery. If a message is not deleted before its visibility timeout expires, another consumer can receive it.
  3. 3. After maxDequeueCount failed attempts, what does the Functions runtime do with a queue message?

    The poison queue (original name plus -poison) keeps messages that could not be processed so people can inspect and replay them.
  4. 4. Why should the handler throw an exception on failure instead of catching everything and returning normally?

    The runtime deletes the message when the function completes successfully. Swallowing an error therefore loses the image without any retry.
  5. 5. What goes into the queue message in a well-designed pipeline?

    Queue messages have a size limit and large payloads are costly. Send a reference and let the worker download the blob using its own identity.

Summary

  • Blob upload raises a BlobCreated event; an Event Grid subscription with a subject filter writes it to a Storage queue; a queue-triggered function does the work.
  • Queues and events give at-least-once delivery, so make handlers idempotent with deterministic outputs and a record of finished work.
  • Let failures throw: the message reappears after the visibility timeout, and after maxDequeueCount attempts moves to the -poison queue.
  • Keep output in a different container from the input, and put references, not payloads, in messages.
  • Alert on backlog and poison-queue length, and replay poison messages after fixing the cause.
  • The stand-in reproduces the queue, retry and poison logic with a virtual clock; real timing and limits come from the Azure services.

Next: Project: Containerized API with a Database.