Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 36 ยท Integration and Deployment

Celery with Django: Setup, on_commit and Best Practices

Set up Celery in a Django project, autodiscover tasks, read settings from Django, avoid the on-commit race with transaction.on_commit, and pass ids instead of objects.

Django and Celery are a classic pair: the web process answers requests quickly, and Celery workers do the slow or scheduled work (sending email, generating PDFs, calling vendors, nightly cleanups). Celery has first-class Django support: no extra package is needed for the integration itself. This lesson shows the project layout, the settings convention, and the three mistakes that account for most Django-plus-Celery bugs.

Django, Celery and a broker do not run on this site. The project files are read-only. The runnable examples are plain-Python stand-ins that reproduce the race with transactions and the "pass an id, reload the object" pattern.

Project layout

TEXT
mysite/
  manage.py
  mysite/
    __init__.py
    celery.py
    settings.py
  orders/
    models.py
    tasks.py

mysite/celery.py creates the app:

Python
import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mysite.settings")

app = Celery("mysite")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

mysite/__init__.py makes sure it loads when Django starts, so @shared_task decorators bind to it:

Python
from .celery import app as celery_app

__all__ = ("celery_app",)

settings.py holds Celery configuration with the CELERY_ prefix (that is what namespace="CELERY" means):

Python
CELERY_BROKER_URL = "redis://localhost:6379/0"
CELERY_RESULT_BACKEND = "redis://localhost:6379/1"
CELERY_TASK_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ["json"]
CELERY_TIMEZONE = "UTC"
CELERY_TASK_TIME_LIMIT = 300
CELERY_TASK_ACKS_LATE = True

autodiscover_tasks() imports a tasks.py module from every installed app, so each app keeps its own tasks next to its models. Run the worker with celery -A mysite worker -l INFO.

Writing tasks with shared_task

Use @shared_task in reusable apps so they do not import the project's app object:

Python
# orders/tasks.py
from celery import shared_task
from .models import Order

@shared_task(bind=True, autoretry_for=(ConnectionError,), retry_backoff=True, max_retries=5)
def send_receipt(self, order_id):
    order = Order.objects.get(pk=order_id)
    mailer.send(order.customer.email, render_receipt(order))

Mistake 1 to avoid: do not pass model instances. Pass the primary key and reload inside the task. An instance is serialized as a snapshot; by the time the worker runs, the row may have changed or been deleted, and JSON cannot serialize model objects anyway. Pass small, JSON-friendly data.

Python Stand-in: pass an id, reload fresh data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The transaction race: use on_commit

Mistake 2 is the most common. In a view:

Python
@transaction.atomic
def checkout(request):
    order = Order.objects.create(...)
    send_receipt.delay(order.pk)          # BUG: the worker may run before COMMIT
    return redirect("thanks")

The message is published immediately, but the transaction commits only when the view finishes. A fast worker runs Order.objects.get(pk=...) and raises DoesNotExist. (Even without atomic, Django's ATOMIC_REQUESTS=True setting wraps every view in a transaction.) The fix:

Python
from django.db import transaction

order = Order.objects.create(...)
transaction.on_commit(lambda: send_receipt.delay(order.pk))

on_commit runs the callback after the outermost transaction commits, and it is dropped if the transaction rolls back (which is exactly right: no receipt for an order that does not exist). Outside a transaction it runs immediately. In tests using TestCase (which wraps tests in a transaction that never commits) use self.captureOnCommitCallbacks(execute=True).

Python Stand-in: queue-before-commit race and its fix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other Django specifics

  • Database connections. Workers are long-lived. Django closes stale connections between requests, but tasks have no request cycle. CONN_MAX_AGE=0 (the default) is simplest; if you persist connections, call django.db.close_old_connections() at task start (Celery's Django fixup does this for you in modern versions).
  • Periodic tasks. Define a beat_schedule in settings, or use django-celery-beat to edit schedules in the admin (a third-party package, installed separately).
  • Results in the database. django-celery-results stores results in your DB and shows them in the admin; Redis is faster for high volume.
  • Settings and secrets. Read the broker URL from the environment (os.environ["CELERY_BROKER_URL"]) so each deployment differs without code changes.
  • Eager mode in tests. CELERY_TASK_ALWAYS_EAGER = True runs tasks inline so tests need no worker; CELERY_TASK_EAGER_PROPAGATES = True re-raises their exceptions. Remember it hides serialization, routing and timing issues, so keep a few integration tests with a real worker or the memory broker (memory://).
  • Logging. Add the task id to log lines with celery.utils.log.get_task_logger(__name__).

A tidy pattern: services and thin tasks

Keep business logic in plain functions and make the task a thin wrapper:

Python
# orders/services.py
def send_receipt_for(order): ...

# orders/tasks.py
@shared_task
def send_receipt(order_id):
    send_receipt_for(Order.objects.get(pk=order_id))

You can unit-test the service without Celery, and call it synchronously in a management command.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does app.config_from_object('django.conf:settings', namespace='CELERY') do?

    The namespace means keys like CELERY_BROKER_URL in settings.py configure the Celery app.
  2. 2. Why should tasks receive an order id instead of an Order instance?

    Pass small JSON-friendly data and reload the object inside the task.
  3. 3. A task occasionally raises DoesNotExist for an object created just before delay(). What is the fix?

    The worker ran before the transaction committed. on_commit publishes after the commit.
  4. 4. What does autodiscover_tasks() do?

    It finds and imports a tasks module in every app listed in INSTALLED_APPS.

Summary

  • Create mysite/celery.py, import it in __init__.py, configure with CELERY_-prefixed Django settings and call autodiscover_tasks().
  • Use @shared_task in apps; pass ids and reload objects inside the task.
  • Publish tasks with transaction.on_commit to avoid the commit race.
  • Keep logic in services, use eager mode only for unit tests, and manage database connections in long-lived workers.

Next: Celery with Flask and FastAPI