Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 36 ยท Getting Started

What Is Celery and Why Use a Task Queue?

Learn what Celery is, which problems a distributed task queue solves, and when to use it. Includes a runnable mini queue that shows the idea without any servers.

Celery is a Python library that runs work outside your main program. You write an ordinary function, mark it as a task, and instead of calling it directly you drop a message into a queue. A separate process called a worker picks the message up and runs the function whenever it has time.

That sounds small, but it changes how you design applications. A web page that must send an email, resize a photo, build a PDF and call a slow payment API can answer the visitor in milliseconds and let the slow parts finish in the background. Celery is the most widely used tool for this job in the Python world, and it works with Django, Flask, FastAPI, or plain scripts.

About the examples on this site. Celery needs a message broker (Redis or RabbitMQ) and long-running worker processes. Neither can run inside a browser page, so the real Celery code in this tutorial is shown as read-only snippets. Whenever an idea can be shown with plain Python, you get a runnable stand-in that is clearly labelled. The stand-ins teach the concepts (queues, messages, workers, retries); they are not Celery itself.

The problem Celery solves

Imagine a signup view that does four things before answering:

  1. Save the user in the database (5 ms).
  2. Send a welcome email through an SMTP server (800 ms, sometimes 5 s).
  3. Generate an avatar image (300 ms).
  4. Notify a CRM over HTTP (400 ms, sometimes times out).

The visitor waits for all of it. If the CRM is down, the signup fails even though the account was created. If a traffic spike brings 1,000 signups per minute, your web workers are busy waiting on email servers instead of serving pages.

The fix is to split the work. The view does the quick, essential part and enqueues the rest. Each queued job is a message: "run send_welcome_email with user_id=42". Workers consume the messages at their own pace, and if one fails it can be retried without the visitor ever noticing.

flowchart LR
    W["Web app<br/>(producer)"] -->|"1. send message"| B["Broker<br/>(queue)"]
    B -->|"2. deliver"| K["Worker"]
    K -->|"3. store result"| R["Result backend"]
    W -.->|"4. read result later"| R

The producer never runs the slow function. It only writes a message and goes on.

What you get with Celery

Plenty of tools can run a function in the background. Celery adds the features you end up needing in production:

  • Reliable delivery. Messages sit in a broker such as Redis or RabbitMQ until a worker acknowledges them.
  • Concurrency. One worker machine can run many tasks at once, using processes, threads or green threads.
  • Retries and error handling. Failed tasks can retry with exponential backoff.
  • Scheduling. The Beat scheduler runs tasks every minute, every night at 02:00, or on the first Monday of a month.
  • Workflows. Chain tasks one after another, run groups in parallel, and run a callback when a group finishes.
  • Routing. Send slow jobs to one queue and urgent jobs to another, each with its own workers.
  • Monitoring. Tools such as Flower and the inspect commands show what is running and what is stuck.
  • Results. A task can return a value that the caller reads later.

When you should use it

Use a task queue when the work is:

  • Slow compared with a web request: reports, video encoding, machine learning inference.
  • Unreliable: calls to third-party services that need retries.
  • Bursty: sending 50,000 newsletters should not slow down your site.
  • Scheduled: nightly cleanups, hourly syncs.
  • Independent of the response: the user does not need the result to see the next page.

Do not reach for Celery when a plain function call is fast enough, when you need an answer immediately in the same request, or when your project is so small that a cron job does the job. Celery adds moving parts (a broker, workers, monitoring), and every moving part can break.

A queue in 25 lines

Before using the real library, build the idea yourself. This stand-in has a list as the "broker", a function registry as the "tasks", and a loop as the "worker". Run it, then read the explanation underneath.

Python Stand-in: a task queue with no Celery

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the order of the printed lines. All three queued lines appear first, followed by "request finished", and only then do the done lines. The producer did not wait for any work.

The stand-in leaves out almost everything that makes a real system hard: the broker survives crashes, many workers read from it at once, messages travel between machines, and a worker that dies midway through a job does not lose it. That is exactly what Celery and a real broker provide.

What the real thing looks like

Here is the same idea written with Celery. This is read-only, because it needs a broker:

Python
# tasks.py
from celery import Celery

app = Celery("demo", broker="redis://localhost:6379/0")

@app.task
def add(x, y):
    return x + y
Python
# somewhere in your web app
from tasks import add

add.delay(2, 3)      # returns immediately with an AsyncResult
Terminal
celery -A tasks worker --loglevel=INFO

Three pieces: the Celery app object with a broker URL, a function decorated with @app.task, and a worker started from the command line. add.delay(2, 3) serializes the call as a message, publishes it to Redis and returns at once.

Producing, consuming and the gap between

The time between "message queued" and "task finished" is where all the interesting problems live. What happens if the worker crashes in the middle? If two workers receive the same message? If the queue grows faster than workers can drain it? If a task needs a result the producer wants to read? The chapters ahead answer each of these. For now keep one mental model: a task call is a message, not a function call.

That has a practical consequence. Arguments must be turned into bytes (usually JSON), so you pass a user_id, not a database object. The worker then loads fresh data itself. You will see why in Task design rules.

How this tutorial is organised

  1. Getting started: architecture, installation, your first app.
  2. Tasks and workers: defining, calling and running tasks.
  3. Brokers and backends: Redis, RabbitMQ, results and routing.
  4. Scheduling and Beat: periodic and cron-style tasks.
  5. Later chapters cover workflows, retries, monitoring, deployment and projects.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does calling add.delay(2, 3) do in Celery?

    delay() only serializes the call into a message and sends it to the broker. A worker runs the function later, possibly on another machine.
  2. 2. Which of these is the best candidate for a background task?

    A welcome email is slow, can fail and is not needed to render the response. The other choices need an immediate answer.
  3. 3. Why do task arguments need to be simple values such as ids and strings?

    A task call becomes a message, so arguments must be serializable (usually JSON). Pass an id and let the worker load the object.
  4. 4. Which component stores messages until a worker is ready for them?

    The broker (Redis, RabbitMQ and others) holds the queue. The result backend stores return values after the task finishes.

Summary

  • Celery runs Python functions in worker processes, outside your web request.
  • A task call is a message sent to a broker; workers consume messages later.
  • It adds retries, scheduling, workflows, routing, monitoring and concurrency on top of the plain queue idea.
  • Use it for slow, unreliable, bursty or scheduled work; avoid it when a plain call or cron job is enough.
  • On this site the real Celery code is read-only; stand-ins run in your browser.

Next: How Celery Works: Producers, Brokers, Workers and Backends