Skip to content
Tutorials Center
My learning

Topics

Lesson 124 of 149 ยท Data and the Web

Background Jobs with Celery and RQ

Learn task queues in Python. Understand brokers, workers and result backends, write Celery tasks with retries, chains and groups, schedule jobs with beat, and compare Celery with RQ.

Imagine a user signs up for your site. You need to save the account, send a welcome e-mail, resize their avatar and notify the sales team. If the web request does all of that, the user stares at a spinner for ten seconds, and if the mail server is down the sign-up fails. The fix is a task queue: the web request just writes down "send welcome mail to [email protected]" and returns immediately. A separate worker process picks the note up and does the slow work in the background.

This pattern powers e-mail, report generation, video processing, web scraping and scheduled maintenance in most large Python systems. The two most common libraries are Celery (powerful, many features) and RQ (small and simple, Redis only). In this lesson you learn the moving parts, then use real Celery in your browser in eager mode, where tasks run immediately in the same process so you can see retries, chains and groups without any servers.

The four moving parts

A task queue has four parts

Producer Your web app calls the task
Broker Holds waiting messages (Redis, RabbitMQ)
Worker Takes a message and runs the code
Result backend Stores the answer (optional)

The producer never waits for the worker.

1 / 2

The broker is a message service. Your app publishes a message that names a task and carries its arguments; workers consume it. Redis and RabbitMQ are the usual brokers. The optional result backend (Redis, a database, and so on) stores return values, so the producer can ask later "is job 7 done, and what did it return?".

Because tasks travel as messages, their arguments must be serializable, which in practice means JSON: pass ids, strings and numbers, not database objects or open files. Pass user_id=42 and let the task load the user itself.

The idea in 20 lines

Before using a library, here is the whole concept with a plain list as the "broker". This is only a model: no persistence, no parallelism.

Python A queue and a worker, by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Celery does exactly this, plus the hard parts: persistent brokers, many worker processes, acknowledgements, retries, rate limits, schedules and monitoring.

Your first Celery tasks

You create a Celery application with a broker URL and decorate functions with @app.task. Calling task.delay(args) sends a message and returns an AsyncResult immediately.

In a real project, the file looks like this and you start a worker from the terminal:

Python
# tasks.py
from celery import Celery

app = Celery("shop", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")

@app.task
def send_welcome(email):
    ...

# in your web app:  send_welcome.delay("[email protected]")
Terminal
celery -A tasks worker --loglevel=info

In the example below we switch on task_always_eager: delay() then runs the function right away in the same process, which is how you unit-test tasks. It still returns a result object with the same interface.

Python Celery in eager mode

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a real worker the producer would see PENDING first and SUCCESS later. Never call result.get() inside another task or a web request on the hot path: waiting for a job defeats the purpose. Return the job id to the client and let it ask again (polling) or use a webhook.

Retries and failure

Background jobs fail for boring reasons: the network blips, the mail server is busy, a rate limit is hit. Celery can retry a task. Inside a task with bind=True, call self.retry(exc=error), optionally with a countdown (seconds to wait). The shortcut is autoretry_for, which retries automatically for the exception types you list, with exponential backoff and random jitter so a thousand failed jobs do not all return at the same second.

Python Retrying a flaky task

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Retries are only safe if the task is idempotent: running it twice must have the same effect as running it once. "Set order 17 to paid" is idempotent. "Add 10 dollars to the balance" is not, so a retry after a half-finished attempt could pay twice. Make jobs idempotent with unique keys, "already done" checks or database constraints.

Related settings you will meet:

  • acks_late=True: the message is acknowledged after the task finishes, so a worker crash re-delivers it (again: needs idempotent tasks).
  • time_limit and soft_time_limit: stop jobs that run too long.
  • rate_limit="10/m": protect an external API.
  • expires: drop a job that is no longer useful if it waited too long.

Chains, groups and chords

Often one job depends on another, or you want many in parallel. Celery calls these canvas primitives. A signature (task.s(args)) is a task call that has not run yet.

  • chain(a.s(), b.s()): run a, feed its result to b.
  • group(a.s(1), a.s(2), ...): run all in parallel.
  • chord(group, callback): run a group, then call the callback with all results.
Python Workflows with chain, group and chord

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A chord needs a result backend that can count finished tasks. That is one more reason to configure a backend when you use workflows.

Scheduled tasks with Celery beat

For periodic jobs ("every night at 2:00, delete expired sessions") Celery ships a scheduler called beat. It is a separate process that publishes messages at the times you configure; ordinary workers run them.

Python
from celery.schedules import crontab

app.conf.beat_schedule = {
    "clean-sessions-nightly": {
        "task": "tasks.clean_sessions",
        "schedule": crontab(hour=2, minute=0),
    },
    "heartbeat-every-30s": {
        "task": "tasks.heartbeat",
        "schedule": 30.0,
    },
}
Terminal
celery -A tasks worker --loglevel=info
celery -A tasks beat --loglevel=info      # run exactly ONE beat process

Run only one beat instance, otherwise every job fires several times. Set app.conf.timezone = "UTC" and keep schedules in UTC to avoid surprises at daylight saving changes.

RQ: the small alternative

RQ (Redis Queue) does one thing with very little ceremony. There is no separate configuration object: you pass a plain function and its arguments to a queue.

Python
from redis import Redis
from rq import Queue, Retry

queue = Queue(connection=Redis())

job = queue.enqueue(send_welcome, "[email protected]", retry=Retry(max=3, interval=[10, 60, 300]))
print(job.id, job.get_status())   # queued, started, finished, failed

# start a worker in the terminal:   rq worker

RQ is easier to learn and to read, but it only supports Redis, has fewer workflow tools and uses a fork-per-job model, so it does not run natively on Windows. Both examples above are read-only on this page, since they need a Redis server.

Celery RQ
Brokers Redis, RabbitMQ, SQS and more Redis only
Scheduling Built in (beat) Add rq-scheduler or similar
Workflows chain, group, chord Dependencies between jobs
Learning curve Steeper, many options Gentle
Good for Large systems, many job types Small to medium apps

If you only need a short follow-up job after a response (log an event, send a quick notification), the framework may already be enough: FastAPI has BackgroundTasks and Starlette has similar helpers. They run in the same process, with no persistence or retries, so a restart loses pending work. Use a real queue for anything important or slow.

Operating a queue

  • Monitor. Watch queue length, task failures and run times. Celery's companion tool Flower shows live workers and tasks.
  • Separate queues. Route slow jobs (video encoding) to their own queue so they cannot block quick ones (password-reset mail).
  • Keep tasks small. Many small tasks are easier to retry and spread than one giant job.
  • Pass ids, not objects, and re-read the data inside the task, so the job sees fresh state.
  • Handle poison messages. A job that always fails should stop after max_retries and go to logs or a dead-letter store for a human to examine.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the broker do in a task queue?

    The broker (Redis, RabbitMQ and others) stores waiting task messages. Workers consume them and run the code.
  2. 2. Which argument is best to pass to a Celery task?

    Task arguments travel as serialized messages (JSON by default). Simple values like ids work, live objects do not, and re-reading the data gives the task current state.
  3. 3. Why should tasks that may be retried be idempotent?

    Retries and late acknowledgements mean the same job can run more than once. Idempotent jobs produce the same final state either way.
  4. 4. What does chord(group(...), callback) do?

    A chord is a group plus a single callback that fires after every group member has finished.
  5. 5. How many Celery beat processes should you run for one schedule?

    Each beat process publishes the scheduled jobs. Running two means every job is queued twice.

Summary

  • A task queue moves slow work out of the request: a producer publishes a message to a broker, a worker executes it, and an optional result backend stores the answer.
  • Celery tasks are plain functions with @app.task; call them with .delay() or .apply_async(countdown=..., queue=...). Arguments must be serializable, so pass ids.
  • task_always_eager runs tasks inline, which is ideal for tests and for learning.
  • Use self.retry or autoretry_for with exponential backoff and jitter, keep tasks idempotent, and set time limits.
  • chain, group and chord build workflows; beat (one instance) runs scheduled jobs.
  • RQ is a simpler Redis-only alternative; framework BackgroundTasks are fine only for small, non-critical follow-ups.

Next: Deploying Python Web Apps.