Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 36 ยท Monitoring and Scaling

Celery Concurrency, Worker Pools and Prefetch

Tune Celery workers: choose prefork, threads, gevent or solo pools, set concurrency, understand prefetch multiplier and max-tasks-per-child, and fix uneven load.

A Celery worker is one program, but it can execute many tasks at the same time. How many, and how, is decided by its pool and its concurrency. Choosing wrongly gives you slow queues, idle CPUs, exhausted memory or a worker that hoards tasks while others sit idle. This lesson explains the options in plain terms and gives a method to pick settings.

Celery does not run on this site. Commands and settings are read-only. The runnable examples are plain-Python stand-ins that model throughput and prefetch so you can see why the numbers behave as they do.

Pools: how tasks run

Select a pool with -P (or --pool) and the number of execution slots with -c (--concurrency):

Terminal
celery -A proj worker -P prefork -c 8 --loglevel=INFO
Pool Model Best for Notes
prefork (default) N child processes CPU-bound work, general use True parallelism, isolation, memory per process. Default concurrency = CPU count
threads N threads in one process I/O-bound, libraries that release the GIL Lower memory; CPU-bound code is limited by the GIL
gevent / eventlet green threads (cooperative) Thousands of concurrent network calls Needs monkey-patching; blocking C libraries stall everything
solo runs tasks inline in the main loop Debugging, tiny containers, one task at a time No concurrency, no time limits

Rules of thumb:

  • CPU-bound (image resizing, parsing, ML inference): prefork with concurrency about equal to the CPU cores.
  • I/O-bound (HTTP calls, sending email): prefork with concurrency of 2 to 4 times the cores, or gevent/threads with 50 to 500 slots. Measure; waiting costs little CPU but memory per prefork process is real.
  • Mixed workload: split into queues and run different workers for each (Queues and routing) instead of one compromise setting.

Concurrency is a ceiling, not a promise. With -c 8, eight tasks can run at the same time. The throughput you get is roughly concurrency / average task seconds.

Python Stand-in: throughput from concurrency

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output shows why: for CPU-bound tasks extra concurrency beyond the core count is wasted (it only adds memory and context switching), while for I/O-bound tasks waiting overlaps and throughput keeps rising.

Prefetch: how many messages a worker grabs

To avoid a network round trip per task, each worker process reserves messages in advance. The number is worker_prefetch_multiplier ร— concurrency (default multiplier 4). A worker with -c 8 holds up to 32 messages locally.

That helps throughput for short tasks but hurts fairness for long ones: a worker may sit on 4 quick tasks queued behind a 10-minute job while another worker is idle. Symptoms: uneven load, tasks waiting although some workers are free.

Tuning:

Python
app.conf.worker_prefetch_multiplier = 1     # fetch one per process at a time
app.conf.task_acks_late = True              # pairs well: ack after the task

or celery -A proj worker -O fair (the default since Celery 4; it avoids handing tasks to children that are busy). For long, uneven tasks use multiplier 1 with acks_late; for millions of tiny tasks keep the default or even raise it.

Note that with acks_late=False prefetched messages are acked when they reach the child process, but still are held by the worker in between; a hard crash loses reserved messages.

Python Stand-in: prefetch hoarding vs fair dispatch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With prefetch 4, the busy worker reserved short tasks it could not start yet, so the idle worker had less to do and the whole job finished later. The gap grows with more short tasks queued behind a long one.

Memory and recycling child processes

Python processes grow: caches, fragmentation, a leak in a C extension. Two protections:

Terminal
celery -A proj worker --max-tasks-per-child=500        # replace a child after 500 tasks
celery -A proj worker --max-memory-per-child=300000    # in KiB: replace after ~300 MB

The child finishes its current task, exits, and the parent forks a fresh one. The price is a little startup cost; the benefit is a bounded memory ceiling. Also useful: loading big models or connections in the worker_process_init signal so that each child sets itself up once, not per task.

Remember that with prefork the parent forks children after import: do not open database connections or sockets at import time, because the children would share them. Open them lazily in each child.

Python Stand-in: recycling keeps memory bounded

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Sizing a worker, step by step

  1. Decide the workload type (CPU or I/O) per task family; separate them into queues.
  2. Start with the pool/concurrency from the table; run under realistic load.
  3. Watch CPU, memory and queue age (Inspecting workers). If CPU is idle and the queue grows, raise concurrency; if memory runs out, lower it or add recycling.
  4. For long tasks set prefetch multiplier 1 and acks_late.
  5. Scale out (more worker containers or machines) once one machine is saturated; the broker coordinates them with no extra setup.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which pool is the default and runs tasks in separate processes?

    prefork forks child processes, giving true parallelism for CPU-bound work.
  2. 2. A worker with -c 4 and the default prefetch multiplier 4 can reserve how many messages?

    The reservation limit is multiplier times concurrency, so 4 x 4 = 16.
  3. 3. Short tasks wait behind a long one while other workers are idle. What helps most?

    A multiplier of 1 stops workers from hoarding messages, so idle workers can take them.
  4. 4. What does --max-tasks-per-child=500 do?

    The child exits after its limit and the parent forks a new one, releasing leaked memory.
  5. 5. For 200 concurrent HTTP calls that mostly wait, which setup is usually most efficient?

    Waiting costs little CPU, so lightweight green threads or threads handle many waits cheaply.

Summary

  • The pool decides how tasks run: prefork (processes), threads, gevent/eventlet (green threads) or solo.
  • Concurrency about equals cores for CPU-bound work and can be much higher for I/O-bound work.
  • Prefetch multiplier times concurrency messages are reserved by each worker; use 1 plus acks_late for long tasks.
  • Recycle children with --max-tasks-per-child or --max-memory-per-child, and open connections lazily per process.
  • Separate different workloads into queues and scale out with more workers.

Next: Queues, routing and priorities