Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 36 ยท Monitoring and Scaling

Celery Queues, Routing and Task Priorities

Route Celery tasks to separate queues with task_routes, dedicate workers to queues, set priorities, and keep slow jobs from blocking urgent ones.

By default every task goes into one queue called celery, and every worker takes from it. That is fine for a demo. In production it creates a classic problem: a nightly export with 50,000 slow tasks fills the queue, and the password-reset email a user is waiting for sits behind all of them. The cure is routing: send different kinds of work to different queues and give each queue its own workers.

Celery and brokers do not run on this site. Celery code is read-only; the runnable examples are plain-Python stand-ins for routing rules and priority ordering.

Define queues and routes

Python
from celery import Celery
from kombu import Queue

app = Celery("proj", broker="redis://localhost:6379/0")

app.conf.task_queues = (
    Queue("default"),
    Queue("emails"),
    Queue("reports"),
)
app.conf.task_default_queue = "default"

app.conf.task_routes = {
    "proj.tasks.send_email": {"queue": "emails"},
    "proj.tasks.export_*":   {"queue": "reports"},     # glob patterns work
    "proj.billing.*":        {"queue": "billing", "routing_key": "billing.high"},
}

Routing happens at the producer when the message is sent, based on the task name. You can also decide per call: export_csv.apply_async(args, queue="reports") or on a signature with .set(queue="reports"). A call-site option wins over task_routes.

task_routes may also be a function or a list of functions for dynamic decisions:

Python
def route_task(name, args, kwargs, options, task=None, **kw):
    if name.endswith("_big"):
        return {"queue": "heavy"}
    return None          # fall through to the next router or the default

Workers choose queues

A worker only consumes the queues you list:

Terminal
celery -A proj worker -Q emails -c 4 -n emails@%h
celery -A proj worker -Q reports -c 2 -n reports@%h
celery -A proj worker -Q default,billing -c 8 -n general@%h

Now the nightly export can back up to a million messages in reports and password resets in emails are untouched. If you forget to start a worker for a queue, its messages wait forever: one of the most common "my task never runs" causes. The --queues list and the task_routes must agree; check with celery -A proj inspect active_queues.

A useful layout is by latency class and resource type: critical (user is waiting), default, bulk (reports, imports), cpu (resizing), io (calls to partners).

Python Stand-in: pattern-based routing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Priorities

Priorities let urgent messages overtake waiting ones inside a queue. Support depends on the broker:

  • RabbitMQ: declare the queue with x-max-priority (for example 10) and send with priority=. Higher numbers are more urgent (0 lowest, up to the max).
  • Redis: Celery emulates priority by splitting a queue into separate lists (queue, queue\x06\x163, ...), with 4 default levels (0, 3, 6, 9). Lower number is more urgent here. It works but is coarse.
  • SQS and others: no priority; use separate queues.
Python
app.conf.task_queue_max_priority = 10        # RabbitMQ
send_email.apply_async(args, priority=9)

# Redis
app.conf.broker_transport_options = {
    "priority_steps": list(range(10)),
    "sep": ":",
    "queue_order_strategy": "priority",
}

The number conventions differ between brokers, which is a good reason to prefer separate queues for things that must never wait, and use priority only for fine ordering within one queue. Prefetching (Concurrency) also blunts priorities: a worker that already reserved 16 low-priority messages will process them before it sees a new high-priority one. Set the prefetch multiplier to 1 when priorities matter.

Python Stand-in: a priority queue serves urgent tasks first

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Operating tips

  • Declare every queue in task_queues so they exist with the right options; by default Celery auto-creates queues it needs (task_create_missing_queues = True), which also means a typo in a queue name silently creates a new, unconsumed queue.
  • Dedicated retry queue. Send retried tasks to queue="retry" served by a smaller worker so failing jobs do not slow fresh work.
  • Name workers (-n name@%h) and keep a table of which worker serves which queue in your runbook.
  • Monitor each queue separately for length and age (Inspecting workers).
  • Rate-limit noisy tasks (rate_limit="10/m") for external APIs. The limit is per worker, not global; for a global limit use a token bucket in Redis.
  • Isolation by queue also isolates failures: a deployment bug in the reports task code can be fixed by restarting only those workers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where is the routing decision made?

    The producer chooses the queue; workers then consume the queues you assign them with -Q.
  2. 2. Your task was routed to queue 'reports' but no worker uses -Q reports. What happens?

    Queues are only drained by workers that consume them, so you need a worker for every queue.
  3. 3. Why can prefetching weaken priorities?

    Reserved messages are processed in the worker's local order, so use a multiplier of 1 when priorities matter.
  4. 4. Which approach is most robust for work that must never wait behind bulk jobs?

    Separate queues and workers give hard isolation; priorities only reorder within one queue.

Summary

  • Define queues with task_queues and map tasks to them with task_routes (patterns or functions), or set queue= per call.
  • Workers consume only the queues given with -Q; every queue needs a worker.
  • Priority support and numbering differ by broker (RabbitMQ max-priority, Redis coarse steps, none on SQS).
  • Isolate by latency class and resource type, use a retry queue, and monitor each queue.
  • Prefer separate queues over priorities for hard guarantees.

Next: Celery with Django