Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 36 ยท Getting Started

How Celery Works: Broker, Workers and Backend

Understand every moving part of a Celery system: producers, brokers, workers, result backends and Beat, and follow one message from call to result.

A Celery system looks complicated in diagrams because it has several boxes, but each box does one small job. Once you know who owns which job, debugging gets much easier: "the task never ran" is a broker or worker problem, "I cannot read the result" is a backend problem, and "it ran twice" is usually an acknowledgement problem.

This lesson walks through the components and then follows a single message through its life.

The five components

Five parts of a Celery system

Producer Your app calls .delay()
Broker Holds the queue
Worker Runs the tasks
Backend Stores results

Beat is the fifth part: it adds scheduled messages.

1 / 2

Producer (client). Any code that calls a task: a Django view, a script, another task. It imports the task, calls delay() or apply_async(), and gets back an AsyncResult handle. It never executes the function.

Broker. A message server that stores messages in queues. Redis and RabbitMQ are the two common choices, Amazon SQS also works. The broker's promise is: if I accepted the message, I will hand it to a consumer.

Worker. A long-running process started with celery worker. It connects to the broker, receives messages, finds the function that matches the task name, runs it, and acknowledges the message. One worker usually runs several tasks at once.

Result backend. An optional store for return values and task states (Redis, a database, and others). Producers use it to ask "is task X done, and what did it return?" Many tasks do not need one.

Beat. A small scheduler process. It does not run tasks itself. At the right times it publishes ordinary task messages to the broker, and workers run them like any other.

One message, start to finish

sequenceDiagram
    participant P as Producer
    participant B as Broker
    participant W as Worker
    participant R as Backend
    P->>B: publish message {task, args, id}
    P-->>P: return AsyncResult(id)
    B->>W: deliver message
    W->>R: state = STARTED
    W->>W: run the function
    W->>R: state = SUCCESS, result
    W->>B: acknowledge (message removed)
    P->>R: result.get() reads the value

The producer's work ends at step one. Everything else happens elsewhere.

A real message is a small structure with headers and a body. Its important fields are:

  • id: a unique UUID. Everything about the task (state, result, logs) is keyed by it.
  • task: the registered name, such as tasks.add. Workers look the function up by this name.
  • args / kwargs: the call arguments, serialized (JSON by default).
  • retries, eta, expires: metadata for retry counting and timing.

A stand-in that mirrors the flow

This runnable version uses the same roles. The broker holds dict messages, the worker has an acks step, and the backend is a dictionary keyed by id. Read how each role touches only its own storage.

Python Stand-in: producer, broker, worker, backend

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Before the worker runs, the backend does not know the id, so the state reads PENDING. That is true in real Celery as well: PENDING really means "I have no information", not "the task is waiting in a queue". A task id that does not exist also shows PENDING.

Where each component can run

Nothing forces these onto one machine. A typical small production setup looks like this:

Component Typical home
Producers Web servers (Django, Flask, FastAPI)
Broker A Redis or RabbitMQ server, or a managed cloud service
Workers One or more separate machines or containers
Result backend The same Redis, or a database
Beat One small process, exactly one instance

Because they communicate only through the broker, you can scale each part independently. More traffic? Add web servers. A backlog of jobs? Add workers. The broker is the single shared piece, which is why its health matters most.

Acknowledgement: the safety net

When a worker receives a message the broker does not delete it right away. The worker must acknowledge (ack) it. By default Celery acknowledges just before the task runs (acks_late=False). That avoids running a task twice but means a crash mid-task loses the message. With acks_late=True the ack happens after the task finishes, so a crash causes redelivery, which means the task may run twice and must be safe to repeat. You will meet this choice again in Task design rules.

Queues, exchanges and routing, briefly

A broker holds one or more queues. By default Celery uses a queue named celery, and every worker consumes from it. Later you can create queues such as emails and reports, route tasks to them by name, and start workers that listen only to certain queues. This gives you priorities and isolation: a flood of report jobs cannot starve the email queue. Routing is covered in Queues and routing.

Common misunderstandings

  • "Celery is a message broker." No. It is a task framework that uses a broker.
  • "The worker is part of my web app." It is a separate process with its own copy of your code. After deploying new code you must restart workers too.
  • "PENDING means queued." It means unknown.
  • "Beat runs the tasks." Beat only publishes messages. Without a worker nothing runs.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which component actually executes the task function?

    Workers receive messages, look up the function by name and run it. The broker only stores messages.
  2. 2. A task id that the backend has never heard of reports which state?

    PENDING is the default for unknown ids, so it can mean either 'waiting' or 'no such task'.
  3. 3. What does Beat do?

    Beat is a scheduler. It sends messages at the configured times and workers execute them.
  4. 4. Why must you restart workers after deploying new task code?

    A running worker imported your modules when it started. It does not notice changed files until it restarts.

Summary

  • Producer calls a task, broker stores the message, worker executes it, backend stores the result, Beat adds timed messages.
  • A message carries an id, a task name and serialized arguments.
  • Components talk only through the broker, so they scale independently.
  • PENDING means "no information", and acknowledgement timing decides whether a crash loses or repeats a task.

Next: Installing Celery and Running Your First Task