Celery Tutorial
Learn Celery step by step: 36 lessons in 9 chapters with examples, quizzes, diagrams and projects.
Start learning →
of lessons completed
Getting Started
- What Is Celery and Why Use a Task Queue? Learn what Celery is, which problems a distributed task queue solves, and when to use it. Includes a runnable mini queue that shows the idea without any servers.
- How Celery Works: Broker, Workers and Backend Understand every moving part of a Celery system: producers, brokers, workers, result backends and Beat, and follow one message from call to result.
- Installing Celery and Running Your First Task Install Celery, start Redis, write your first task, run a worker and call the task with delay(). A step-by-step guide with a runnable stand-in of the same flow.
- Project Layout, App Configuration and Autodiscovery Organise a Celery project: one app module, a settings file, autodiscovered tasks, environment-based configuration and the most useful settings, with runnable stand-ins.
Tasks and Workers
- Defining Tasks with @app.task and Task Options Learn how to define Celery tasks: the decorator, task names, bind=True and self.request, per-task options, custom base classes and the names that workers use.
- Calling Tasks: delay, apply_async, countdown and eta Learn every way to call a Celery task: delay, apply_async, countdown, eta, expires, queue and priority, apply for local runs, and how to test with eager mode.
- Task Results, States and AsyncResult Understand Celery task states (PENDING, STARTED, SUCCESS, FAILURE, RETRY), how to read results with AsyncResult, when to ignore results and how to avoid blocking.
- Running Workers: Pools, Concurrency and Prefetch Start and tune Celery workers: prefork, threads, gevent and solo pools, concurrency, prefetch multiplier, autoscaling, graceful shutdown and the options you set in production.
- Task Design Rules: Idempotent, Small and Safe Design Celery tasks that survive retries and duplicates: idempotency, passing ids instead of objects, small arguments, safe serialization, acks_late and time limits.
Brokers and Backends
- Message Brokers Compared: Redis, RabbitMQ and SQS Compare the message brokers Celery supports, understand delivery guarantees, what a broker must do, and how to choose between Redis, RabbitMQ, SQS and others.
- Using Redis as a Celery Broker: URLs and Visibility Timeout Configure Redis as a Celery broker: URL formats, databases, Sentinel, TLS, the visibility timeout that causes duplicate tasks, memory and persistence concerns.
- Using RabbitMQ: Exchanges, Queues and Routing Keys Set up RabbitMQ as a Celery broker and understand AMQP exchanges, queues, bindings and routing keys, durability, priorities, virtual hosts and the management UI.
- Result Backends: Redis, Database, RPC and More Choose and configure a Celery result backend: Redis, SQLAlchemy/Django databases, RPC, S3 and others, with expiry, extended results, size limits and what to store.
- Queues and Routing: Sending Tasks Where They Belong Split work into queues with task_routes, route by task name or glob, start workers per queue with -Q, set default queues, and keep slow jobs from starving urgent ones.
Scheduling and Beat
- Periodic Tasks with Celery Beat Run tasks on a schedule with Celery Beat: beat_schedule, interval schedules, add_periodic_task, starting beat, the schedule file, and how Beat and workers fit together.
- Crontab and Solar Schedules in Celery Use Celery crontab schedules to run tasks at exact times: fields, ranges, steps, day-of-week and month rules, timezones, solar schedules and a runnable cron matcher.
- Dynamic Schedules: Database-Backed Periodic Tasks Change Celery Beat schedules at runtime without restarting: custom schedulers, django-celery-beat, SQLAlchemy and Redis schedulers, and how Beat detects changes.
- Running Beat in Production: Single Instance, Locks and Overlap Run Celery Beat safely: exactly one instance, supervision, lock-based overlap prevention, catching up after downtime, monitoring missed runs and high-availability options.
Workflows: Chains and Groups
- Celery Signatures: Partials, Immutability and Options Learn Celery signatures, the building block of workflows: task.s(), partial arguments, immutable signatures with .si(), options and how arguments are merged.
- Celery Chains: Run Tasks in Sequence Build Celery chains with chain() and the pipe operator, understand how results flow between tasks, handle failures, and read results from the last task.
- Celery Groups: Run Tasks in Parallel Use Celery groups to run many tasks in parallel, collect results with GroupResult, handle partial failures, and combine groups with chains.
- Celery Chords: Parallel Tasks With a Callback Learn Celery chords: run a group in parallel then call one callback with all results. Covers the header, body, backend needs, error behavior and pitfalls.
- Celery map, starmap and chunks: Batch Many Items Learn the Celery canvas helpers map, starmap and chunks to process large lists efficiently, choose batch sizes, and combine them with groups and chords.
Retries and Reliability
- Celery Retries and Exponential Backoff Make Celery tasks resilient: self.retry, autoretry_for, retry_backoff, jitter, max_retries, custom countdowns, and how to avoid retry storms.
- Celery acks_late and Idempotent Tasks Understand Celery message acknowledgement, acks_late, reject_on_worker_lost and visibility timeouts, and design idempotent tasks that are safe to run twice.
- Celery Time Limits, Expiry and Revoking Tasks Stop runaway Celery tasks with soft and hard time limits, expire stale messages, revoke queued or running tasks, and clean up safely with SoftTimeLimitExceeded.
- Celery Failure Handling, Callbacks and Dead Letters Handle Celery task failures: link and link_error callbacks, Task.on_failure and on_retry hooks, signals, dead-letter patterns and alerting for exhausted retries.
Monitoring and Scaling
- Inspecting Celery Workers and Monitoring With Flower Monitor Celery with the inspect and control APIs, the celery CLI, task events and the Flower dashboard. Learn what to watch: queue length, task age and failures.
- Celery Concurrency, Worker Pools and Prefetch Tune Celery workers: choose prefork, threads, gevent or solo pools, set concurrency, understand prefetch multiplier and max-tasks-per-child, and fix uneven load.
- Celery Queues, Routing and Task Priorities Route Celery tasks to separate queues with task_routes, dedicate workers to queues, set priorities, and keep slow jobs from blocking urgent ones.
Integration and Deployment
- Celery with Django: Setup, on_commit and Best Practices Set up Celery in a Django project, autodiscover tasks, read settings from Django, avoid the on-commit race with transaction.on_commit, and pass ids instead of objects.
- Celery with Flask and FastAPI: Task Status APIs Integrate Celery with Flask (app factory, ContextTask) and FastAPI (202 Accepted, task status endpoints), and choose between BackgroundTasks and Celery.
- Deploying Celery With Docker, Compose and systemd Run Celery workers and beat in production with Docker Compose or systemd: environment config, one beat only, graceful shutdown, health checks and zero-downtime deploys.
Projects and Practice
- Project: An Image Thumbnail Pipeline With Celery Build a Celery pipeline for image uploads: validate, create thumbnails in parallel, collect results with a chord, record status and handle failures and retries.
- Project: A Report Fan-Out Service With Celery Design a Celery service that builds a large report by splitting it into chunks, running them in parallel, tracking progress, throttling an upstream API, and merging results.
- Celery Best Practices, Pitfalls and Practice Problems A production checklist for Celery: small idempotent tasks, JSON payloads, timeouts, retries, queues, monitoring, plus practice exercises and interview-style questions.