Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 36 ยท Integration and Deployment

Deploying Celery With Docker, Compose and systemd

Run Celery workers and beat in production with Docker Compose or systemd: environment config, one beat only, graceful shutdown, health checks and zero-downtime deploys.

Developing with celery worker in a terminal is easy. Production is about different questions: how do workers start after a reboot, how are they restarted during a deploy without losing work, how do you make sure exactly one beat scheduler runs, and where does configuration live? This lesson answers them with two common setups: Docker Compose and systemd.

Docker, systemd, Celery and a broker do not run on this site. All configuration files are read-only. The runnable examples are plain-Python stand-ins for environment-based configuration and the warm-shutdown behavior.

Principles before tools

  1. One image, many roles. The web app, the worker and beat run the same code with different commands.
  2. Configuration from the environment. Broker URL, result backend and secrets come from environment variables, not from files committed to git.
  3. Stateless workers. All state lives in the broker, database and result backend, so you can start or kill workers freely.
  4. Exactly one beat. Two beat processes schedule every periodic task twice.
  5. Graceful stop. Let running tasks finish before a worker exits.

Docker Compose

YAML
# compose.yaml
services:
  redis:
    image: redis:7
    restart: unless-stopped

  web:
    build: .
    command: gunicorn mysite.wsgi:application --bind 0.0.0.0:8000
    env_file: .env
    depends_on: [redis]
    ports: ["8000:8000"]

  worker:
    build: .
    command: celery -A mysite worker -l INFO -Q default,emails --concurrency=4
    env_file: .env
    depends_on: [redis]
    restart: unless-stopped
    stop_grace_period: 120s        # time for running tasks to finish
    healthcheck:
      test: ["CMD-SHELL", "celery -A mysite inspect ping -d celery@$$HOSTNAME --timeout 5"]
      interval: 60s
      timeout: 10s
      retries: 3

  beat:
    build: .
    command: celery -A mysite beat -l INFO --schedule=/data/celerybeat-schedule
    env_file: .env
    volumes: [beat-data:/data]
    depends_on: [redis]
    restart: unless-stopped        # exactly one replica

volumes:
  beat-data:

Details worth knowing:

  • Scaling workers: docker compose up -d --scale worker=3. Do not scale beat.
  • Signals. Docker sends SIGTERM on stop. Celery treats it as a warm shutdown: stop taking new tasks, finish running ones, exit. stop_grace_period must be longer than your longest task or Docker sends SIGKILL after the default 10 seconds. Kill-9 loses prefetched messages, unless acks_late is on.
  • PID 1. Use the exec form of CMD/command (as above, no shell wrapper) so Celery receives the signal. The worker's own children are handled by Celery.
  • Run as a non-root user in the Dockerfile (USER app). Celery warns when running as root with pickle enabled; stay on the JSON serializer.
  • Beat state. Beat writes its schedule file; mount a volume so a restart does not forget when tasks last ran, or use a database scheduler.
  • Separate images are not needed; separate containers with different command lines are.
Python Stand-in: build configuration from the environment

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

systemd (virtual machines and bare metal)

Without containers, systemd supervises the processes. A worker unit:

INI
# /etc/systemd/system/celery-worker.service
[Unit]
Description=Celery worker
After=network.target redis.service

[Service]
Type=simple
User=celery
Group=celery
WorkingDirectory=/srv/mysite
EnvironmentFile=/etc/mysite/env
ExecStart=/srv/mysite/venv/bin/celery -A mysite worker -l INFO --concurrency=4 --statedb=/var/lib/celery/worker.state
ExecStop=/bin/kill -s TERM $MAINPID
TimeoutStopSec=300
Restart=on-failure
RestartSec=5
KillMode=mixed
RuntimeDirectory=celery
StateDirectory=celery

[Install]
WantedBy=multi-user.target

and a similar celery-beat.service with ExecStart=... celery -A mysite beat -l INFO --schedule=/var/lib/celery/beat-schedule. Enable with systemctl enable --now celery-worker celery-beat. TimeoutStopSec plays the same role as stop_grace_period; KillMode=mixed sends SIGTERM to the main process and SIGKILL to leftovers only after the timeout. Logs go to journalctl -u celery-worker -f.

For many workers use a template unit ([email protected]) and start celery-worker@1, @2; set the node name with -n worker%i@%h so each worker has a unique name.

Zero-downtime deploys

  1. Build the new image or code with the same task names and compatible message formats. A worker running old code may receive a message produced by new code and vice versa during the rollout, so change task signatures in two steps: first accept both old and new arguments, deploy, then switch producers.
  2. Start new workers, then send SIGTERM to old ones (warm shutdown). They finish current tasks and exit; with acks_late any interrupted tasks return to the queue.
  3. Restart beat last (one instance only); a few seconds without beat just delays periodic tasks.
  4. Keep the broker and result backend up across deploys. Never flush Redis (FLUSHALL) in a deploy script: it deletes queued jobs.
Python Stand-in: warm shutdown finishes running work

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Production checklist

  • Broker with persistence (Redis appendonly yes, RabbitMQ durable queues) and monitoring.
  • Separate workers per queue with their own concurrency (Queues and routing, Concurrency).
  • --max-tasks-per-child or memory limits to bound leaks.
  • Container memory limits larger than concurrency times the per-task peak.
  • JSON serializer; never accept pickle from untrusted producers.
  • A heartbeat check and alerting (Inspecting workers).
  • One beat, with its schedule file on durable storage.
  • Secrets from the environment or a secret manager; Flower behind authentication.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How many beat processes should run for one Celery application?

    Each beat publishes periodic tasks, so two beats would schedule every task twice.
  2. 2. What does SIGTERM do to a Celery worker?

    Warm shutdown lets in-flight tasks finish; give the container or unit a long enough stop timeout.
  3. 3. Why is stop_grace_period (or TimeoutStopSec) important?

    If the grace period is shorter than running tasks, the orchestrator kills the worker before they finish.
  4. 4. Where should the broker URL and secrets come from in production?

    Environment-based configuration keeps secrets out of the code and lets each deployment differ.

Summary

  • Use one image with separate commands for web, worker and beat; configure through the environment.
  • Run exactly one beat, with its schedule file on a volume or durable disk.
  • Rely on SIGTERM warm shutdown and set stop_grace_period or TimeoutStopSec above the longest task.
  • Use health checks, JSON serializer, non-root users and persistent brokers.
  • Deploy in compatible steps so old and new code can share the queue during rollout.

Next: Project: a thumbnail pipeline