Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 36 ยท Monitoring and Scaling

Inspecting Celery Workers and Monitoring With Flower

Monitor Celery with the inspect and control APIs, the celery CLI, task events and the Flower dashboard. Learn what to watch: queue length, task age and failures.

A Celery system has moving parts you cannot see: messages waiting in a broker, worker processes busy with long tasks, a task that has been "running" for two hours. Before you scale or debug anything you need to look. Celery ships with a remote-control API and a command line, and the community tool Flower adds a web dashboard on top. This lesson covers all three and the few numbers that actually matter.

Celery, brokers, workers and Flower do not run on this site. Their commands and code are read-only. The runnable examples are plain-Python stand-ins that compute the same kinds of numbers (queue age, failure rate, worker load) from sample data.

The celery command line

All of these assume the app is called proj (-A proj):

Terminal
celery -A proj status                 # which workers answer
celery -A proj inspect active         # tasks running right now
celery -A proj inspect reserved       # tasks prefetched by workers, not started
celery -A proj inspect scheduled      # ETA/countdown tasks held by workers
celery -A proj inspect registered     # task names each worker knows
celery -A proj inspect stats          # pool size, uptime, total tasks per name
celery -A proj inspect ping
celery -A proj events                 # live curses monitor
celery -A proj purge                  # DELETE all waiting messages (careful!)

inspect and control are broadcast commands: the CLI sends a message to every worker's control queue and collects the replies for a short timeout (one second by default; raise it with --timeout). A busy or frozen worker may not answer in time, so a missing reply is a hint, not proof.

The same API is available from Python:

Python
inspector = app.control.inspect(timeout=2)
inspector.active()       # {'worker1@host': [{'id': ..., 'name': 'tasks.export', 'time_start': ...}]}
inspector.stats()
inspector.active_queues()   # which queues each worker consumes

Control commands

Python
app.control.pool_grow(2)                       # add 2 pool processes
app.control.pool_shrink(1)
app.control.add_consumer("emails", reply=True) # start consuming a new queue
app.control.cancel_consumer("emails")          # stop consuming it (drain a queue)
app.control.rate_limit("tasks.send_sms", "10/m")
app.control.shutdown()                         # warm shutdown of every worker

rate_limit, time_limit and pool_* change running workers without a restart, which is useful during an incident. Those runtime changes are lost at restart, so also change the real configuration.

Task events

Workers can emit events (task received, started, succeeded, failed, retried; worker online, heartbeat). Monitoring tools consume them. Enable with the -E flag or worker_send_task_events = True:

Terminal
celery -A proj worker -E --loglevel=INFO

Events add a little overhead, so turn them on when you monitor with Flower or custom consumers. They are best-effort (UDP-like fire and forget semantics over the broker), so treat them as telemetry rather than an audit log.

Flower

Flower is a web dashboard and REST API built on the events and the control API. It is a separate package:

Terminal
pip install flower
celery -A proj flower --port=5555 --basic-auth=admin:change-me

It shows workers with their load and pool size, a task list with state, arguments, runtime and traceback, graphs of tasks per name, queue lengths (with a Redis or RabbitMQ broker API) and buttons for revoke, rate limit and pool resize. Treat Flower like an admin console:

  • Put it behind authentication and a firewall. It can revoke tasks and shut workers down.
  • Task arguments are displayed; do not pass secrets as task arguments.
  • Flower keeps history in memory (limited by --max_tasks); use --persistent=True with a state file or ship events elsewhere for longer retention.

The numbers that matter

  1. Queue length per queue: how many messages wait. With Redis, LLEN queue_name; with RabbitMQ, the management API.
  2. Oldest message age (queue latency): more useful than length, because 10,000 tiny tasks may be fine and 3 stuck ones not.
  3. Task runtime percentiles per task name: catches a slow query before it hits time limits.
  4. Failure and retry rate per task name.
  5. Worker count and heartbeat: a missing worker means lost capacity.
  6. Worker memory (see Concurrency and prefetch).

Add simple alerts: queue age above N seconds, failure rate above N percent, zero workers.

Python Stand-in: compute queue age, runtimes and failure rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Health checks in practice

  • Liveness for containers: celery -A proj inspect ping -d celery@$HOSTNAME exits non-zero if that worker does not reply. Use it as a Kubernetes/Docker health check, but give it a generous timeout so a busy worker is not killed.
  • Heartbeat task: schedule a tiny task with beat every minute that writes a timestamp; alert if the timestamp is old. It checks the whole pipeline: beat, broker, worker, result path.
  • Log in JSON with the task id and name so you can follow one job across services.
Python Stand-in: a heartbeat watchdog

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does celery inspect active list?

    inspect active shows tasks being run now; reserved shows prefetched ones that have not started.
  2. 2. Why might celery inspect not list a worker that is actually running?

    A missing reply is only a hint; raise the timeout before concluding that a worker is gone.
  3. 3. Which metric usually warns earliest that workers cannot keep up?

    Queue latency shows how long work waits; it rises as soon as capacity is short.
  4. 4. How should Flower be deployed?

    Flower is an admin console with powerful controls, so protect it like one.

Summary

  • celery status, inspect and control use broadcast messages to ask workers what they are doing and to change them live.
  • Events (-E) feed dashboards; Flower adds a web UI and REST API and must be secured.
  • Watch queue length and especially oldest message age, runtime percentiles, failure rate, worker count and memory.
  • Add a heartbeat task and container health checks to detect silent failure.

Next: Concurrency, pools and prefetch