Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 36 ยท Brokers and Backends

Queues and Routing: Sending Tasks Where They Belong

Split work into queues with task_routes, route by task name or glob, start workers per queue with -Q, set default queues, and keep slow jobs from starving urgent ones.

By default every task goes into one queue named celery, and every worker takes tasks from it. That is fine until a flood of slow jobs (a nightly export of 200,000 rows) delays an urgent one (a password-reset email). The fix is routing: put different kinds of tasks in different queues and decide which workers serve which queue.

flowchart LR
    A["web app"] --> R{"routing rules"}
    R -->|"emails.*"| Q1["queue: emails"]
    R -->|"reports.*"| Q2["queue: reports"]
    R -->|"everything else"| Q3["queue: default"]
    Q1 --> W1["workers -Q emails<br/>concurrency 8"]
    Q2 --> W2["workers -Q reports<br/>concurrency 2"]
    Q3 --> W3["workers -Q default"]

Isolation: a backlog in reports cannot delay emails, because different workers serve them.

Declaring queues and a default

Python
from kombu import Queue

app.conf.task_queues = (
    Queue("default"),
    Queue("emails"),
    Queue("reports"),
)
app.conf.task_default_queue = "default"

If you do not list queues at all, Celery creates them on demand (task_create_missing_queues=True), which is convenient but means a typo in a queue name silently creates a new, unserved queue. When a queue exists but no worker listens to it, its messages wait forever, so listing your queues explicitly and turning off auto-creation catches mistakes early:

Python
app.conf.task_create_missing_queues = False

Choosing a queue per call

The most direct way is the queue option:

Python
send_email.apply_async((42,), queue="emails")

It works but scatters queue names through your code. A better way is to keep the decision in one place with task_routes.

task_routes

Python
app.conf.task_routes = {
    "shop.tasks.send_email": {"queue": "emails"},
    "shop.tasks.send_*": {"queue": "emails"},          # glob pattern
    "reports.*": {"queue": "reports", "priority": 2},
    "re:^billing\\..*": {"queue": "billing"},           # regular expression
}

Keys are task names. Glob patterns (*) and regular expressions (prefix re:) are allowed; the first matching rule wins when keys are matched in order. Values are options passed to apply_async, so queue, routing_key, priority and others work. A call-time option such as queue= overrides the route.

You can also use a router function when the decision depends on arguments:

Python
def route_task(name, args, kwargs, options, task=None, **kw):
    if name.startswith("reports.") and kwargs.get("big"):
        return {"queue": "reports_big"}
    if name.startswith("emails."):
        return {"queue": "emails"}

app.conf.task_routes = (route_task,)

The function returns a dict (or None to pass to the next router). This stand-in resolves routes the same way, with globs, a regex, a function and the default queue:

Python Stand-in: resolving task_routes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Starting workers for queues

Workers listen to the queues you give with -Q:

Terminal
celery -A shop worker -Q emails -c 8 -n emails@%h
celery -A shop worker -Q reports -c 2 -n reports@%h
celery -A shop worker -Q default,emails -n general@%h

A worker without -Q consumes from every queue in task_queues (or just celery). You can change what a running worker consumes without restarting it:

Terminal
celery -A shop control add_consumer emails
celery -A shop control cancel_consumer reports

This is useful for draining a backlog by temporarily adding capacity.

Designing queues

Good reasons to create a queue:

  • Different latency needs: user-facing tasks (emails, notifications) versus batch work.
  • Different resource needs: tasks needing lots of memory or a GPU run on special machines.
  • Different concurrency or pool: I/O-bound HTTP calls with gevent in one queue, CPU work with prefork in another.
  • Rate limiting or isolation of a flaky dependency: if a partner API is slow, only its queue backs up.
  • Tenants or priorities: separate queues for paying customers.

Avoid creating a queue per task type by reflex. Each queue needs at least one worker, so dozens of queues become an operations burden. Three to six queues cover most systems.

Priority: queues beat per-message priority

Per-message priority works on RabbitMQ and is emulated on Redis, but it has limits (see Message brokers compared). The simplest robust approach is separate queues with separate workers:

Python
app.conf.task_routes = {
    "auth.*": {"queue": "high"},
    "newsletter.*": {"queue": "low"},
}

Run more workers on high, or give a mixed worker -Q high,low and accept that its prefetch can mix them. For strict priority you can run a worker on high only.

Routing keys and exchanges (advanced)

On AMQP, queue declarations bind an exchange and a routing key; task_routes can give routing_key and exchange:

Python
from kombu import Exchange, Queue

tasks_exchange = Exchange("tasks", type="topic")
app.conf.task_queues = (
    Queue("video", tasks_exchange, routing_key="video.#"),
    Queue("email", tasks_exchange, routing_key="email.#"),
)
app.conf.task_routes = {
    "media.convert": {"exchange": "tasks", "routing_key": "video.convert"},
}

This matches the topic rules you practised in the RabbitMQ lesson. With Redis, exchanges are only emulated, so keep to simple queue names.

Troubleshooting routing

  • Tasks pile up in a queue and nothing runs: no worker is consuming it. Run celery -A shop inspect active_queues to see which queue each worker reads.
  • A task runs on the wrong worker: check which route matched. A glob like shop.* may catch more than intended; order and specificity matter.
  • Typos: Queue("emial") and queue="email" are different queues. Turn off task_create_missing_queues in production.
  • Task registered on the worker but not routed: a worker serving -Q emails still needs to import the task, because routing and registration are separate.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A queue named reports has 5,000 waiting messages and nothing is running. What is the most likely cause?

    Messages wait until a worker listens to that queue. Check with inspect active_queues.
  2. 2. Which option has priority when both task_routes and the call specify a queue?

    Call-time options override routing rules.
  3. 3. How do you make a worker consume only the emails queue?

    -Q (--queues) lists the queues a worker consumes.
  4. 4. Why is it risky to leave task_create_missing_queues enabled in production?

    Auto-creation hides mistakes. A message in an unserved queue just waits.

Summary

  • Queues isolate workloads; routes decide where each task goes, -Q decides who serves each queue.
  • Use task_routes with names, globs, regexes or a function; call-time queue= overrides them.
  • A small number of well-chosen queues (by latency, resources and failure isolation) beats many.
  • Separate queues are a stronger priority mechanism than per-message priority.

Next: Periodic Tasks with Celery Beat