Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 36 ยท Getting Started

Project Layout, App Configuration and Autodiscovery

Organise a Celery project: one app module, a settings file, autodiscovered tasks, environment-based configuration and the most useful settings, with runnable stand-ins.

A single tasks.py is fine for a first try. A real project has dozens of tasks spread across feature folders, different settings in development and production, and secrets that must not be committed. This lesson shows a layout that scales and the settings you will touch most.

TEXT
myproject/
  myproject/
    __init__.py
    celery_app.py        # creates the Celery app
    settings.py          # your configuration
    billing/
      __init__.py
      tasks.py           # tasks for billing
    notifications/
      __init__.py
      tasks.py
  pyproject.toml

The idea: one module creates the Celery instance, and every feature keeps its own tasks.py that imports that instance. The app finds them automatically.

Python
# myproject/celery_app.py
from celery import Celery

app = Celery("myproject")
app.config_from_object("myproject.settings", namespace="CELERY")
app.autodiscover_tasks()          # looks for tasks.py in installed packages
Python
# myproject/__init__.py
from .celery_app import app as celery_app

__all__ = ("celery_app",)

Importing the app in the package __init__ makes sure it loads when Django or your framework starts, so @shared_task decorators bind to it.

Python
# myproject/billing/tasks.py
from celery import shared_task

@shared_task
def charge_invoice(invoice_id):
    ...

@shared_task creates a task that is not tied to a particular app object, which keeps reusable modules independent from your project. Use @app.task when the module already imports the app and is not meant to be reused.

Start the worker with celery -A myproject worker -l INFO. Because -A myproject points at a package, Celery looks for an attribute named celery or app there, or for a celery_app/celery module.

How config_from_object works

config_from_object reads settings from a module, a class or a dict. With namespace="CELERY", only names that start with CELERY_ are read, and the prefix is dropped. That lets you share one settings file with Django without clashes:

Python
# myproject/settings.py
CELERY_BROKER_URL = "redis://localhost:6379/0"
CELERY_RESULT_BACKEND = "redis://localhost:6379/1"
CELERY_TASK_SERIALIZER = "json"
CELERY_TIMEZONE = "UTC"

Celery 4 and later use lowercase setting names internally (broker_url, result_backend, task_serializer). The namespace mechanism maps CELERY_BROKER_URL to broker_url for you. This stand-in shows exactly that translation:

Python Stand-in: reading settings with a namespace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two unrelated settings are skipped. If you forget the namespace in a real project, mixed-case names such as BROKER_URL from old Celery versions can silently be ignored, so check the worker banner to confirm the broker it actually connected to.

Autodiscovery

app.autodiscover_tasks() imports a tasks module from each package it is told about. Without arguments in Django it uses INSTALLED_APPS. Without Django you pass the packages yourself:

Python
app.autodiscover_tasks(["myproject.billing", "myproject.notifications"])

Discovery is lazy: it happens when the worker starts, not when you import the app. That is why a task you define in a different file name (for example jobs.py) is not found; either rename it to tasks.py or tell Celery:

Python
app.autodiscover_tasks(["myproject.billing"], related_name="jobs")
# or list modules explicitly
app.conf.imports = ["myproject.billing.jobs"]

The stand-in below builds two tiny in-memory packages and imports their tasks modules by name, which is what autodiscovery does behind the scenes.

Python Stand-in: autodiscovering tasks modules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The worker's banner lists exactly such names. If a name is missing there, discovery did not reach that module.

Configuration by environment

Never hard-code a production broker password. Read values from environment variables:

Python
# settings.py
import os

CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL", "redis://localhost:6379/0")
CELERY_RESULT_BACKEND = os.environ.get("CELERY_RESULT_BACKEND", "redis://localhost:6379/1")

Then set them per environment: a .env file in development, the platform's secret store in production. You can also override at the command line, which is handy for one-off experiments:

Terminal
celery -A myproject worker -l INFO --concurrency=2 -Q default

Settings you will use first

Setting What it does
broker_url Where messages are sent.
result_backend Where results and states are stored.
task_serializer / accept_content How messages are encoded and which formats workers accept (keep JSON).
timezone, enable_utc Time zone for scheduling. UTC is the safest.
task_default_queue Name of the default queue (celery).
task_track_started Report a STARTED state, not just PENDING/SUCCESS.
task_time_limit, task_soft_time_limit Hard and soft run-time limits in seconds.
worker_prefetch_multiplier How many messages each process reserves in advance.
result_expires How long results are kept (one day by default).
broker_connection_retry_on_startup Keep retrying if the broker is down at start.

You can print the effective configuration of a running worker with celery -A myproject inspect conf, and read a single value in code with app.conf.task_serializer.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does namespace="CELERY" do in config_from_object?

    The namespace lets Celery settings live next to other settings (such as Django's) in one file. CELERY_BROKER_URL becomes broker_url.
  2. 2. When should you use @shared_task instead of @app.task?

    shared_task binds to whichever app is current, which suits reusable packages.
  3. 3. Your task lives in billing/jobs.py and the worker does not list it. What fixes this?

    By default autodiscovery imports modules named tasks. Other file names must be configured.
  4. 4. Where should the production broker password live?

    Keep secrets out of source control; read them from the environment.

Summary

  • Create the Celery app in one module, give each feature its own tasks.py, and import the app in the package __init__.
  • config_from_object(..., namespace="CELERY") maps CELERY_* names to Celery's lowercase settings.
  • autodiscover_tasks() imports tasks modules lazily when the worker starts; check the banner's task list.
  • Read secrets from the environment and know the handful of settings that matter first.

Next: Defining Tasks