Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 36 ยท Projects and Practice

Project: An Image Thumbnail Pipeline With Celery

Build a Celery pipeline for image uploads: validate, create thumbnails in parallel, collect results with a chord, record status and handle failures and retries.

In this project you design an upload pipeline like the ones behind photo sites: a user uploads an image, the request returns immediately, and workers validate it, create several thumbnail sizes in parallel, and finally mark the image as ready. You will use everything from the workflow chapter: signatures, a chain, a group inside a chord, plus retries, idempotency and failure callbacks.

Celery, a broker, workers and image libraries do not run on this site. The Celery code is read-only. The runnable part is a small plain-Python workflow engine stand-in (about 40 lines) that runs the same chain/group/chord shape on fake "images" (just dictionaries), so you can test the pipeline logic itself.

The requirements

  1. POST /images stores the original file and returns 202 with an image id. The request never waits for processing.
  2. A worker validates the file (type, size, decodable).
  3. Three thumbnails (small, medium, large) are created in parallel.
  4. When all three exist, the image is marked ready with the thumbnail paths. Any permanent failure marks it failed with a reason.
  5. Processing a message twice must not create duplicates or corrupt data.
  6. Slow, CPU-heavy resizing must not block quick tasks such as notifications.

Design

flowchart LR
    A["POST /images"] --> B["validate_image"]
    B --> C["group: make_thumbnail x3"]
    C --> D["finalize_image (chord body)"]
    B -. "link_error" .-> E["mark_failed"]
    D --> F["status = ready"]

The chain is validate | chord(group(thumbnails), finalize); any failure ends in mark_failed.

Routing: validate_image and finalize_image are cheap and go to the default queue; make_thumbnail is CPU-bound and goes to a cpu queue served by a prefork worker with concurrency equal to the cores (Concurrency).

The Celery code (read-only)

Python
# tasks.py
from celery import Celery, chain, chord, group

app = Celery("photos", broker="redis://redis:6379/0", backend="redis://redis:6379/1")
app.conf.task_routes = {"photos.make_thumbnail": {"queue": "cpu"}}
app.conf.task_acks_late = True
app.conf.worker_prefetch_multiplier = 1

SIZES = {"small": 128, "medium": 512, "large": 1280}

@app.task(name="photos.validate_image", bind=True, autoretry_for=(OSError,),
          retry_backoff=True, max_retries=3)
def validate_image(self, image_id):
    image = storage.load(image_id)
    if image.size_bytes > 20_000_000 or image.format not in {"JPEG", "PNG", "WEBP"}:
        raise ValueError("unsupported image")          # permanent: not retried
    return image_id

@app.task(name="photos.make_thumbnail", bind=True, autoretry_for=(OSError,),
          retry_backoff=True, max_retries=3, soft_time_limit=60, time_limit=90)
def make_thumbnail(self, image_id, label):
    path = f"thumbs/{image_id}/{label}.jpg"
    if storage.exists(path):                           # idempotent: safe to repeat
        return {label: path}
    storage.save(path, resize(storage.load(image_id), SIZES[label]))
    return {label: path}

@app.task(name="photos.finalize_image")
def finalize_image(results, image_id):
    paths = {k: v for part in results for k, v in part.items()}
    Image.objects.filter(pk=image_id).update(status="ready", thumbnails=paths)
    return paths

@app.task(name="photos.mark_failed")
def mark_failed(failed_task_id, image_id):
    reason = repr(app.AsyncResult(failed_task_id).result)
    Image.objects.filter(pk=image_id).update(status="failed", error=reason)

def start_pipeline(image_id):
    thumbs = group(make_thumbnail.si(image_id, label) for label in SIZES)
    workflow = validate_image.si(image_id) | chord(thumbs, finalize_image.s(image_id))
    return workflow.apply_async(link_error=mark_failed.s(image_id))

Walk through the design choices:

  • .si() on make_thumbnail because the chord header tasks should not receive the validation result as an extra argument.
  • finalize_image.s(image_id) is a mutable body: the list of thumbnail results is prepended, so it runs as finalize_image(results, image_id).
  • The idempotency check (storage.exists) makes duplicate deliveries harmless. acks_late plus prefetch 1 gives at-least-once delivery without hoarding.
  • ValueError is not in autoretry_for, so a bad file fails immediately and triggers mark_failed; OSError (a storage blip) is retried with backoff.
  • The view only calls start_pipeline(image.id) after the database commit (transaction.on_commit in Django, see Django integration).

The runnable stand-in

This engine runs steps in order, runs a group's members one after another (a real worker pool would run them in parallel), passes the list to the chord body, and calls the error callback on any failure. It does not model queues or timing; it tests the logic.

Python Stand-in: the thumbnail pipeline end to end

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check the output: image 1 is ready with three paths, image 2 is failed, and the duplicate run for image 1 did not add files.

Testing the real pipeline

  1. Unit tests call the task function directly (make_thumbnail.run(...), or the plain function) with a fake storage.
  2. Workflow tests use task_always_eager=True with task_eager_propagates=True to run chains inline. Chords need a real or in-memory backend; with eager mode Celery runs the chord inline too, but verify with an integration test.
  3. Integration test with a real worker started by the celery.contrib.pytest fixtures or Docker Compose, a Redis container and a handful of real small images.
Python Stand-in: a test that asserts idempotency and failure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Extensions to try

  • Add a fourth step after the chord: notify_user.si(user_id) with a short expires so a late notice is dropped.
  • Report progress with self.update_state and show a progress bar.
  • Move storage to an object store and pass keys, never file bytes, through the broker.
  • Rate-limit make_thumbnail if the image library is memory-hungry, and use --max-memory-per-child.
  • Write an on_failure hook that stores dead letters, then a management command that replays them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why are the make_thumbnail signatures created with .si()?

    An immutable signature ignores the parent's return value, so make_thumbnail receives only its own arguments.
  2. 2. How does the pipeline survive a duplicate delivery of make_thumbnail?

    An idempotency check in the task makes a second run harmless.
  3. 3. What triggers mark_failed?

    link_error runs when a task in the workflow fails for good, such as the ValueError for an unsupported format.
  4. 4. Why send make_thumbnail to its own cpu queue?

    Separate queues give isolation and let you set concurrency to match the CPU cores.

Summary

  • The pipeline is validate | chord(group(thumbnails), finalize) with link_error for failures.
  • Immutable signatures keep header tasks independent; the chord body receives the list of results.
  • Idempotent tasks, acks_late and retries with backoff make it safe and resilient.
  • Heavy work lives in its own queue; messages carry ids and keys, not files.
  • Test logic with plain functions and eager mode, then confirm with a real worker.

Next: Project: a report fan-out service