Skip to content
Tutorials Center
My learning

Topics

Lesson 63 of 77 ยท Deployment and Production

Docker, Celery and CI for Django

Package Django in Docker, run Postgres, Redis, Celery workers and beat with Compose, and test every push with GitHub Actions. Includes a live task-queue stand-in.

A Django site in production is rarely one process. There is a web server running your views, a database, often a cache, and a second kind of process that does slow work in the background: sending email, resizing images, building reports. You also want every change to be tested automatically before it can reach those machines. Three tools cover that: Docker to package the app, Celery to run background tasks, and a CI pipeline to check each push.

This lesson shows the files you would write for each of them. Docker, Redis, Postgres and GitHub are not available inside a browser tab, so those files are read-only. The ideas behind them, though, are plain Python, and the live examples below run a miniature task queue and a Django app that hands work to it.

If Docker itself is new to you, the Docker tutorial explains images and containers from scratch. The previous lesson, Deploying Django, covers the production checklist this lesson builds on.

The shape of the system

Every container runs the same image with a different command. That is the key idea: you build once, then start the web server, the worker and the scheduler from it.

flowchart LR
    U["Browser"] --> W["web<br/>gunicorn"]
    W --> DB[("db<br/>Postgres")]
    W -->|"task message"| R[("redis<br/>broker")]
    R --> K["worker<br/>celery worker"]
    B["beat<br/>celery beat"] -->|"schedule"| R
    K --> DB

The web process queues work in Redis; workers pick it up. Beat only adds scheduled messages to the same queue.

The web process never waits for the slow job. It drops a small message into Redis and responds immediately. A worker reads the message later and does the work. If the worker crashes, the message is not lost and the web site stays up.

A Dockerfile for Django

A good Django image installs dependencies first (so Docker can cache that layer), runs as a normal user, and starts Gunicorn:

DOCKERFILE
# syntax=docker/dockerfile:1
FROM python:3.13-slim AS base

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

WORKDIR /app

# Install dependencies first: this layer is cached until requirements change
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# Collect static files at build time (WhiteNoise serves them)
RUN SECRET_KEY=build-only python manage.py collectstatic --noinput

RUN useradd --create-home appuser
USER appuser

EXPOSE 8000
CMD ["gunicorn", "config.wsgi:application", "--bind", "0.0.0.0:8000", "--workers", "3"]

Points worth noticing:

  • PYTHONUNBUFFERED=1 makes log lines appear immediately in docker logs instead of waiting in a buffer.
  • collectstatic needs a settings module that imports, so a throwaway secret is passed only for that one command. Real secrets must never be baked into an image.
  • USER appuser means a bug in your app does not run as root inside the container.
  • Add a .dockerignore listing .git, .venv, __pycache__, .env and db.sqlite3, so they never enter the build. See build context and .dockerignore.

For smaller images, use a multi-stage build that compiles wheels in one stage and copies only the result into the final stage.

Compose: web, worker, beat, Postgres and Redis

Docker Compose describes the whole group of containers in one file. Notice that web, worker and beat share one build and one set of environment variables:

YAML
x-django: &django
  build: .
  env_file: .env
  depends_on:
    db:
      condition: service_healthy
    redis:
      condition: service_started

services:
  web:
    <<: *django
    command: gunicorn config.wsgi:application --bind 0.0.0.0:8000
    ports:
      - "8000:8000"

  worker:
    <<: *django
    command: celery -A config worker --loglevel=info --concurrency=2

  beat:
    <<: *django
    command: celery -A config beat --loglevel=info

  db:
    image: postgres:17
    environment:
      POSTGRES_DB: app
      POSTGRES_USER: app
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d app"]
      interval: 5s
      retries: 10

  redis:
    image: redis:7

volumes:
  pgdata:

The .env file holds the settings your Django project reads from the environment:

TEXT
DJANGO_SECRET_KEY=change-me
DJANGO_DEBUG=0
ALLOWED_HOSTS=example.com
DATABASE_URL=postgres://app:secret@db:5432/app
REDIS_URL=redis://redis:6379/0

Inside the Compose network, containers reach each other by service name, so the database host is db and the broker host is redis, not localhost. Run docker compose up --build, then apply migrations once with docker compose run --rm web python manage.py migrate. See Compose for a full stack for more patterns.

Only one beat container should ever run. Two schedulers would queue every periodic task twice.

Settings that come from the environment

A container image must work unchanged on your laptop, in CI and in production. The only thing that changes is the environment. So settings.py reads values from environment variables, which is the second live example below. A typical block:

Python
import os

SECRET_KEY = os.environ["DJANGO_SECRET_KEY"]
DEBUG = os.environ.get("DJANGO_DEBUG", "0") == "1"
ALLOWED_HOSTS = os.environ.get("ALLOWED_HOSTS", "").split(",")

Use os.environ["NAME"] (not .get) for values that must exist, so a missing secret stops the process at start rather than failing later in a confusing way.

Celery in a Django project

Celery is the most widely used task queue for Django. Three small pieces connect it to a project. First the Celery app, in config/celery.py:

Python
import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")

app = Celery("config")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

Then import it in config/__init__.py so it loads with Django:

Python
from .celery import app as celery_app

__all__ = ("celery_app",)

And settings, all prefixed with CELERY_ because of the namespace:

Python
CELERY_BROKER_URL = os.environ["REDIS_URL"]
CELERY_TASK_ACKS_LATE = True          # re-deliver if a worker dies mid-task
CELERY_TASK_TIME_LIMIT = 300          # hard stop after 5 minutes

CELERY_BEAT_SCHEDULE = {
    "nightly-report": {
        "task": "reports.tasks.build_report",
        "schedule": 60 * 60 * 24,     # every 24 hours, in seconds
    },
}

Then you write tasks in a tasks.py file inside any app. autodiscover_tasks() finds them:

Python
from celery import shared_task

@shared_task(autoretry_for=(ConnectionError,), retry_backoff=True, max_retries=5)
def send_welcome_email(user_id):
    ...

You call a task with .delay(), which only queues a message and returns at once:

Python
send_welcome_email.delay(user.pk)

Rules for good tasks

  • Pass ids, not objects. The message is serialized to JSON. Send user.pk, and load the user inside the task so you see the current data.
  • Make tasks idempotent. Because of retries and re-delivery, a task can run twice. Running it twice must be harmless (for example, check a "welcome email sent" flag first).
  • Queue after the commit. If you call .delay() inside a transaction, the worker may start before the data is committed and fail to find the row. Use transaction.on_commit, shown below.

Django 6.0 added a built-in tasks API (django.tasks) for defining and enqueuing background work. It provides the interface and development backends; running tasks reliably in production still needs a worker backend such as Celery. The concepts in this lesson (queue, worker, retry, idempotency) apply to both.

A task queue you can run here

Celery needs Redis and a separate worker process, which a browser cannot start. The next example is a stand-in that behaves like a tiny Celery: a decorator, .delay(), a broker holding messages, a worker that retries failures with a growing delay, and a "beat" that schedules periodic work. Time is simulated with a counter, so the run is instant and the output is identical every time.

Python A miniature task queue with retries and beat

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the log from top to bottom. flaky_report fails at t=0, waits 2 seconds, fails again, waits 4 seconds (the delay doubles each time: exponential backoff) and succeeds at t=6. always_broken runs out of retries and ends in FAILURE. Beat is nothing more than "enqueue this task every N seconds"; the worker does not care who queued the message.

Queue work only after the commit

Here is the transaction trap in live code. The view creates a signup and queues a welcome email. With transaction.on_commit the email is queued only if the database transaction really commits. The settings are read from environment variables, as in the Compose file above, and a /healthz view shows the kind of endpoint a load balancer or container health check calls.

Python Env-driven settings and on_commit task dispatch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The failed signup rolled back its row and queued nothing, while the successful one queued exactly one message. Without on_commit, the failed request would have queued a task for a user that no longer exists. The last line shows ALLOWED_HOSTS doing its job: a request for an unknown host gets 400.

Testing every push with GitHub Actions

Continuous integration (CI) runs your checks on a clean machine each time someone pushes. If the workflow fails, you find out before the code is merged. Save this as .github/workflows/ci.yml:

YAML
name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:17
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: app
        ports:
          - 5432:5432
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s
          --health-retries 10
    env:
      DJANGO_SECRET_KEY: ci-only-secret
      DJANGO_DEBUG: "0"
      ALLOWED_HOSTS: localhost
      DATABASE_URL: postgres://postgres:postgres@localhost:5432/app
      REDIS_URL: redis://localhost:6379/0
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
          cache: pip
      - run: pip install -r requirements.txt
      - run: python manage.py check --deploy --fail-level WARNING
      - run: python manage.py makemigrations --check --dry-run
      - run: python manage.py test

Each step has a clear job:

  • check --deploy fails the build if a security setting is missing. (Your CI settings may need a few extra production values for this to be clean.)
  • makemigrations --check --dry-run fails if you changed a model but forgot to commit the migration, a very common mistake.
  • manage.py test runs your test suite against real Postgres, the same database as production. Testing on SQLite while deploying on Postgres hides bugs.

In tests, set Celery to run tasks immediately (CELERY_TASK_ALWAYS_EAGER = True) so you do not need a broker, or test the task function by calling it directly like any Python function.

A second job can build the Docker image and push it to a registry only when tests pass and the branch is main. See CI pipelines for Docker.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In the Compose file, why do web, worker and beat all use the same image?

    Building one image and starting it with different commands guarantees the three processes run identical code and avoids version drift between them.
  2. 2. Why should you call a task with transaction.on_commit(lambda: task.delay(obj.pk)) instead of calling .delay() directly inside an atomic block?

    Without on_commit the message can reach a worker before the data is visible, and a rolled-back transaction would still leave a queued task pointing at a missing row.
  3. 3. What does exponential backoff mean for task retries?

    Growing delays give an overloaded service time to recover instead of hammering it with instant retries.
  4. 4. What does python manage.py makemigrations --check --dry-run catch in CI?

    It exits with an error if Django would create a new migration, which means someone changed a model without committing the matching migration file.
  5. 5. Which Celery task argument is the better choice?

    Messages are serialized and may wait in the queue. A primary key is small and safe to serialize, and the task loads fresh data when it runs.

Summary

  • Build one Docker image and run it as web, worker and beat with different commands; Postgres and Redis are separate services.
  • Read secrets and connection settings from environment variables so the image is identical everywhere.
  • Celery tasks are queued with .delay(), should receive ids rather than objects, and must be safe to run twice.
  • Use transaction.on_commit to queue a task only after the data is committed, and run only one beat scheduler.
  • A CI workflow should run check --deploy, makemigrations --check and the tests against the same database engine you use in production.
  • Retries with exponential backoff protect struggling services; the stand-in queue above shows the mechanics without Redis.

Next lesson: Performance and Scaling.