Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 36 ยท Brokers and Backends

Result Backends: Redis, Database, RPC and More

Choose and configure a Celery result backend: Redis, SQLAlchemy/Django databases, RPC, S3 and others, with expiry, extended results, size limits and what to store.

The result backend is where Celery stores a task's state and return value so that someone else can read them later. It is completely separate from the broker: the broker moves messages to workers, the backend collects results from workers.

You need a backend only if you:

  • call result.get(), result.state or result.ready(),
  • use workflows that pass values along (chains, groups, chords), or
  • want a record of failures and tracebacks.

If your tasks are fire-and-forget (send an email, clear a cache), skip the backend entirely and set ignore_result=True. That is faster and removes one dependency.

Configuring a backend

Python
app.conf.result_backend = "redis://localhost:6379/1"

The URL scheme selects the implementation:

Backend URL example Good for
Redis redis://host:6379/1 Fast, short-lived results; the most common choice.
Database (SQLAlchemy) db+postgresql://user:pw@host/db Results you want to query and keep.
Django ORM django-db (package django-celery-results) Django projects that want results in admin.
Django cache django-cache Results stored in the Django cache framework.
RPC rpc:// Results returned as messages to the caller's own queue.
Memcached cache+memcached://host:11211/ Simple, volatile storage.
MongoDB mongodb://host:27017/celery When you already run Mongo.
S3 s3:// (needs celery[s3]) Large results, cheap storage.
File system file:///var/results Single machine tests.
Disabled not set Fire-and-forget systems.

For the database backend, install the extra (pip install "celery[sqlalchemy]"), and Celery creates two tables, celery_taskmeta and celery_tasksetmeta, on first use.

What gets stored

For every finished task the backend stores a small record. With result_extended=True you also get the task name, arguments, queue and worker, which makes debugging much easier:

Python
app.conf.result_extended = True

This stand-in models the record structure and the extended fields:

Python Stand-in: what a stored result looks like

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Result expiry

Results do not live forever. result_expires (default 86400 seconds, one day) defines how long a result is kept. With Redis, Celery uses key expiry, so no cleanup is needed. With a database backend, a built-in periodic task called celery.backend_cleanup deletes old rows; it needs Beat to be running.

Python
app.conf.result_expires = 3600          # one hour
app.conf.result_expires = None          # never expire (be careful with memory/disk)

Why expiry matters: a million tasks per day with a 10 KB result each is 10 GB per day. Combine short expiry, small results and ignore_result for tasks that do not need values.

The stand-in below models a store with a time to live and a clock you advance yourself:

Python Stand-in: result expiry (TTL) on a simulated clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Once expired, the id looks exactly like one that never existed. If a user returns to a status page the next day, show a friendly "result no longer available" message.

The RPC backend

rpc:// is special. Instead of storing results, the worker sends the result as a message back to the process that called the task, using a reply queue. It is very fast and needs no extra server, but:

  • Only the client that sent the task can read the result, and only once.
  • Results are lost if that client restarts.
  • Workflows that need a stored result (chords) will not work.

It suits short-lived scripts and RPC-style calls. For web apps with many processes, use Redis or a database.

Choosing a backend

  • Redis: default choice. Watch memory; set a short result_expires.
  • Database: when you need to search, report on or keep results (and can afford the write load). Watch table growth and the cleanup task.
  • S3 or similar: for big payloads, but remember that results are loaded into memory on get().
  • Nothing: when you never read results. Fewer moving parts, faster tasks.

A hybrid is often best: do not return big data from tasks at all. Write the output to your own table or object storage and return a small reference:

Python
@app.task
def build_report(user_id):
    key = make_pdf_and_upload(user_id)       # stored by you
    return {"s3_key": key}                    # tiny result

Settings worth knowing

Setting Purpose
result_backend URL of the backend.
result_expires Seconds until results are deleted.
result_extended Store name, args and worker with the result.
result_serializer Format of stored results (keep json).
result_compression Compress large results (e.g. gzip).
task_ignore_result Do not store results by default.
task_track_started Store STARTED when a task begins.
result_backend_transport_options Backend-specific options, e.g. Redis retry_policy.
result_chord_retry_interval How often chords poll for member results.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Do you need a result backend to run tasks?

    Tasks run without a backend. You need one to read results or use chords and similar features.
  2. 2. What does the rpc:// backend do?

    RPC results travel back to the caller via a reply queue, so only that client can read them, and only once.
  3. 3. How are old results removed from a database backend?

    With key-expiry backends like Redis the store expires keys itself; database backends rely on the cleanup task.
  4. 4. What is a good way to handle a task that produces a 200 MB file?

    Results are for small values. Keep big data in object storage or your own tables.

Summary

  • The backend is separate from the broker and optional; use ignore_result=True for fire-and-forget tasks.
  • Redis is the common default; databases suit queryable history; RPC is single-reader and volatile.
  • Results expire (default one day); database backends depend on the cleanup task and Beat.
  • Keep results tiny: return ids and keys, not payloads.

Next: Queues and Routing: Sending Tasks Where They Belong