Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 36 ยท Tasks and Workers

Running Workers: Pools, Concurrency and Prefetch

Start and tune Celery workers: prefork, threads, gevent and solo pools, concurrency, prefetch multiplier, autoscaling, graceful shutdown and the options you set in production.

A worker is just a process that connects to the broker and runs tasks, but how it runs them has a big impact on throughput and reliability. This lesson explains the pool types, the knobs that matter and the way a worker stops.

Starting a worker

Terminal
celery -A myproject worker --loglevel=INFO

Useful options you will meet quickly:

Option Meaning
-A, --app The application module.
-l, --loglevel DEBUG, INFO, WARNING, ERROR.
-c, --concurrency Number of concurrent execution slots. Defaults to the CPU core count.
-P, --pool Execution pool: prefork, threads, gevent, eventlet, solo.
-Q, --queues Comma-separated queues to consume from.
-n, --hostname Worker name, e.g. worker1@%h. Needed to run several on one machine.
-O Optimization profile, e.g. -O fair.
--max-tasks-per-child Recycle a child process after N tasks.
--max-memory-per-child Recycle a child after it exceeds N KiB.
--autoscale=10,3 Scale between 3 and 10 child processes.

Every worker needs a unique node name. Two workers on one host without -n worker1@%h and -n worker2@%h would clash, and remote control commands could not tell them apart.

Choosing a pool

The pool decides how tasks run in parallel.

Pick the pool for the job

prefork Processes. CPU-bound work.
threads Threads. Simple I/O work.
gevent Thousands of slow I/O calls.
solo One task at a time.

prefork is the default and the safest choice.

1 / 2
  • prefork (default): the main worker process forks child processes. Each task runs in its own process, so Python's GIL is not a limit and a crash of one child does not kill the others. It is the right default for CPU-bound or mixed work. Memory use is higher because each child imports your code.
  • threads: tasks run in threads of one process. Cheaper per task, but the GIL limits CPU-bound code. Good for modest I/O-bound loads.
  • gevent / eventlet: green threads that switch while waiting for the network. A single worker can have hundreds or thousands of concurrent slots (-c 500). Only helpful when tasks mostly wait on I/O, and every library must be cooperative (monkey-patched) or it blocks the whole worker.
  • solo: runs tasks inline in the main process, one at a time. Useful for debugging, containers that scale by replicas, and on platforms where prefork is not supported (Celery's prefork is not officially supported on Windows; use solo or threads for development there).

Rule of thumb: for CPU-heavy tasks, concurrency equals the number of cores. For I/O-heavy tasks with prefork, you can go higher (two or three times the cores), or switch to gevent and set a large number.

Prefetch: why one slow task blocks others

Each worker process reserves messages ahead of time so it does not wait for the network between tasks. The amount is worker_prefetch_multiplier * concurrency (default multiplier 4). With four processes, the worker holds up to 16 unacknowledged messages.

That is fast, but it can be unfair. Imagine one long task and several short ones in the same queue. If a process reserves the long task and three short ones, those three wait behind it, even while other processes sit idle. Setting worker_prefetch_multiplier = 1 (or starting with -O fair) reserves only one message per slot, which is fairer for tasks of very different length.

The next stand-in simulates exactly this: two processes, a mix of long and short jobs, run with prefetch 4 and prefetch 1. Time is simulated, there is no sleeping.

Python Stand-in: prefetch and fairness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With prefetch 4, three small emails are stuck behind the long report on one process while the other process finishes its own queue early. With prefetch 1 the idle process keeps taking short jobs, so all emails finish much sooner. The report takes the same time either way. For uniform short tasks, a high prefetch is actually better, because it hides broker latency.

Two related settings:

  • task_acks_late=True with prefetch: late acknowledgement means reserved messages stay unacknowledged until done, which is what makes redelivery after a crash possible.
  • worker_disable_rate_limits: turns off rate-limit machinery if you never use it.

Memory and long-running children

Python processes that run for days can grow. Two guards recycle them:

Terminal
celery -A myproject worker --max-tasks-per-child=500 --max-memory-per-child=300000

When a child hits the limit it finishes the current task and is replaced by a fresh process. This is a pragmatic fix for slow leaks in image or data libraries.

Autoscaling

Terminal
celery -A myproject worker --autoscale=8,2

The pool grows to at most 8 processes when the queue is busy and shrinks to 2 when idle. It works within one machine; scaling machines is a deployment problem (covered later).

Stopping a worker gracefully

Signals matter in production:

Signal Effect
SIGTERM Warm shutdown: stop taking new tasks, let running tasks finish, then exit.
SIGQUIT Cold shutdown: terminate as soon as possible.
SIGINT (Ctrl+C) Same as warm shutdown; pressed twice, it becomes cold.
SIGHUP Restart the worker.
SIGUSR1 Dump the stack of all threads to the log.

Containers and process managers send SIGTERM and then, after a grace period, SIGKILL. Make sure the grace period (for example terminationGracePeriodSeconds in Kubernetes, or stopwaitsecs in Supervisor) is longer than your longest task, or the task is killed mid-run. With default early acknowledgement that task is then lost; with acks_late=True it is redelivered and must be idempotent.

Inspecting a running worker

Terminal
celery -A myproject inspect active      # tasks running now
celery -A myproject inspect reserved    # prefetched but not started
celery -A myproject inspect stats       # pool size, uptime, counters
celery -A myproject inspect registered  # task names it knows
celery -A myproject control shutdown    # warm shutdown of all workers

These commands use the broker to broadcast to all workers and wait for replies. The reserved list is how you see prefetch in action.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which pool is the default and the usual choice for CPU-bound tasks?

    prefork uses separate processes, so the GIL does not limit CPU-bound work and a crashing task does not take the worker down.
  2. 2. A queue mixes one 5-minute task with many 1-second tasks, and the quick ones wait too long. What is the usual fix?

    High prefetch lets a process reserve short tasks that then queue behind the long one. Prefetch 1 or separate queues avoid that.
  3. 3. What does SIGTERM do to a Celery worker?

    SIGTERM triggers a warm shutdown. SIGQUIT is the cold shutdown.
  4. 4. Why should two workers on one machine use different --hostname values?

    Node names must be unique so inspect and control commands can address each worker.

Summary

  • Start workers with celery -A app worker -l INFO; give each node a unique name.
  • prefork for CPU work, threads or gevent for I/O waits, solo for debugging and one-task-per-container setups.
  • Concurrency sets slots; prefetch sets how many messages each slot reserves. Lower it when task lengths vary widely.
  • Recycle children with --max-tasks-per-child and --max-memory-per-child.
  • SIGTERM is a warm shutdown, so give containers a grace period longer than your longest task.

Next: Task Design Rules: Idempotent, Small and Safe