Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 36 ยท Projects and Practice

Project: A Report Fan-Out Service With Celery

Design a Celery service that builds a large report by splitting it into chunks, running them in parallel, tracking progress, throttling an upstream API, and merging results.

Reports are the textbook Celery job: the data set is large, the work divides cleanly, and users should not stare at a spinner tied to a web request. In this project you build a monthly sales report that covers 50,000 orders. The service splits the orders into batches, processes batches in parallel on several workers, shows live progress, gently calls a rate-limited exchange-rate API, and merges everything into one file. You will practice chunks and batching, chords, queues, time limits and cancellation.

Celery, brokers and workers do not run on this site. The Celery code is read-only. The runnable part is a plain-Python stand-in that splits data, processes batches, tracks progress and merges results, so the logic is testable here.

Requirements

  1. The user presses "Generate report" and immediately gets a report id; the page polls for progress.
  2. Orders are split into batches of 500; batches run in parallel.
  3. Each batch converts amounts to the report currency using a rate-limited API (10 calls per minute per worker); rates are cached so a batch does one call per currency, not per order.
  4. Progress shows done/total batches.
  5. The user can cancel a running report.
  6. A failed batch is retried; a permanently failed batch fails the report with a clear reason.
  7. The final merge produces one CSV path and a summary.

Design

flowchart TD
    U["POST /reports"] --> S["start_report: create report row"]
    S --> SP["split_orders: compute batch ranges"]
    SP --> G["group: process_batch x N"]
    G --> M["merge_report (chord body)"]
    G -. "progress counter in Redis" .-> P["GET /reports/id"]
    M --> D["status = done, file = path"]

Batches pass only id ranges; each worker loads its own rows from the database.

Key decisions:

  • Pass ranges, not rows. A batch task receives (report_id, first_id, last_id) and queries the database itself. Messages stay tiny.
  • Progress counter. Each finished batch runs INCR report:<id>:done in Redis. The status endpoint compares it with the total. No polling of Celery internals needed.
  • Cancel flag. SET report:<id>:cancelled 1; each batch checks it before starting. Cooperative cancellation is safer than revoke(terminate=True).
  • Rate limit. Celery's rate_limit="10/m" is per worker. For a global limit across all workers, use a Redis token bucket or a single dedicated worker (concurrency=1) on a fx queue that fetches exchange rates. The batch tasks read rates from a cache.
  • Cheap retries. Batch tasks are idempotent: they write their partial result to reports/<id>/part-<n>.csv, overwriting any earlier attempt.

The Celery code (read-only)

Python
from celery import Celery, chord, group
import redis

app = Celery("reports", broker="redis://redis:6379/0", backend="redis://redis:6379/1")
store = redis.Redis(host="redis", decode_responses=True)
BATCH = 500

@app.task(name="reports.start")
def start_report(report_id, month):
    first, last = order_id_range(month)
    ranges = [(lo, min(lo + BATCH - 1, last)) for lo in range(first, last + 1, BATCH)]
    store.set(f"report:{report_id}:total", len(ranges))
    store.set(f"report:{report_id}:done", 0)
    header = group(process_batch.s(report_id, lo, hi, n) for n, (lo, hi) in enumerate(ranges))
    return chord(header)(merge_report.s(report_id)).id

app.conf.task_routes = {"reports.batch": {"queue": "batch"}}

@app.task(name="reports.batch", bind=True,
          autoretry_for=(ConnectionError,), retry_backoff=True, max_retries=4,
          soft_time_limit=240, time_limit=300)
def process_batch(self, report_id, lo, hi, part):
    if store.get(f"report:{report_id}:cancelled"):
        return None                                   # cooperative cancel
    rows = load_orders(lo, hi)
    rates = cached_rates(["EUR", "GBP", "USD"])       # one API call per currency, cached
    path = f"reports/{report_id}/part-{part}.csv"
    write_csv(path, [(r.id, r.amount * rates[r.currency]) for r in rows])
    store.incr(f"report:{report_id}:done")
    return path

@app.task(name="reports.merge")
def merge_report(parts, report_id):
    if store.get(f"report:{report_id}:cancelled"):
        return "cancelled"
    final = f"reports/{report_id}/report.csv"
    concat_files([p for p in parts if p], final)
    return final

The API layer exposes POST /reports (starts start_report.delay), GET /reports/{id} (reads the counters and the final state), and DELETE /reports/{id} (sets the cancel flag). Add a failure callback that writes failed into the report row, and send batches to their own queue served by a worker tuned for the workload (Concurrency).

The stand-in

Python Stand-in: split, process, count progress and merge

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Throttling an upstream API

Celery's rate_limit is enforced per worker process group, so three workers with 10/m allow 30 calls a minute in total. If the partner allows 10 per minute globally, use a shared counter. This stand-in implements a token bucket with a fake clock so you can see the rule:

Python Stand-in: a token bucket for a global rate limit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When a task is denied, call raise self.retry(countdown=6) instead of sleeping, so the worker slot is free for other work.

Extensions

  • Store rates in a database table with a daily refresh task scheduled by beat.
  • Send the user an email when the report finishes with a link that expires.
  • Add a PDF summary as a second chord body step.
  • Put a hard cap of one running report per user using a Redis lock with an expiry.
  • Compare batch sizes 100, 500 and 5,000 on your hardware and chart the total time.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do batch tasks receive id ranges rather than the order rows?

    Small messages are cheaper, and the worker reads current data from the database.
  2. 2. How does the status endpoint know the progress?

    A shared counter is simple, cheap and independent of Celery internals.
  3. 3. Why is a Celery rate_limit not enough for a global partner limit?

    Each worker enforces the limit separately; a shared token bucket or a single-worker queue gives a global limit.
  4. 4. What is the safest way to cancel a running report?

    Cooperative cancellation lets batches stop cleanly without killing processes that may be doing other work.

Summary

  • Split big jobs into batch tasks that receive ranges, run them in a group, and merge in a chord body.
  • Track progress with a shared counter and support cooperative cancellation with a flag.
  • Retry batch failures with backoff, make them idempotent and give them time limits.
  • Use caches and a global token bucket for rate-limited upstream APIs.
  • Isolate batch work on its own queue with tuned workers.

Next: Celery best practices and practice problems