Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 36 ยท Retries and Reliability

Celery Time Limits, Expiry and Revoking Tasks

Stop runaway Celery tasks with soft and hard time limits, expire stale messages, revoke queued or running tasks, and clean up safely with SoftTimeLimitExceeded.

Without limits, one stuck task can occupy a worker slot forever: a database query that never returns, an HTTP call to a server that accepts the connection and then says nothing. A queue backed up for an hour may also contain messages nobody wants any more. Celery gives you three tools: time limits (stop a running task), expiry (drop a message that waited too long) and revoke (cancel on demand).

Celery does not run on this site. Celery code is read-only; the runnable examples are plain-Python stand-ins that model limits and expiry with a fake clock, so the logic is real but nothing is killed.

Soft and hard time limits

Python
from celery.exceptions import SoftTimeLimitExceeded

@app.task(soft_time_limit=50, time_limit=60)
def build_report(report_id):
    try:
        rows = load_rows(report_id)
        return render(rows)
    except SoftTimeLimitExceeded:
        mark_report_failed(report_id, "took too long")
        raise
Limit What happens Can the task react?
soft_time_limit Celery raises SoftTimeLimitExceeded inside the task Yes: catch it, clean up, exit
time_limit (hard) The worker kills the child process running the task No: no cleanup runs

Use both: the soft limit a little below the hard one gives the task a chance to roll back, release locks and record a failure before the process is killed. Defaults can be set app-wide with task_soft_time_limit and task_time_limit, and overridden per call: build_report.apply_async(args, soft_time_limit=10).

Notes:

  • Limits apply to the prefork pool (the default). The solo and threads pools cannot kill a running function reliably, and the gevent/eventlet pools support only soft behavior for cooperative code.
  • The clock starts when the task starts running, not when it was queued.
  • Limits protect worker capacity but they are not a substitute for timeouts on network calls. Always set timeout= on requests calls and database statement timeouts; they fail faster and more cleanly.
  • A hard kill means acks_late tasks may be redelivered. Consider whether the task should be retried or marked failed.
Python Stand-in: soft limit gives the task time to clean up

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Expiry: drop stale messages

A reminder that is delivered a day late may be useless or even harmful. Give the message a lifetime:

Python
send_flash_sale_email.apply_async(args=[user_id], expires=600)           # seconds
send_flash_sale_email.apply_async(args=[user_id], expires=datetime(...))  # absolute time

If a worker picks up the message after it expired, it discards the task and marks it REVOKED. You can also set a default expires in the task decorator. Expiry is checked when the worker receives the message, so it does not interrupt a running task; use time limits for that.

Expiry is excellent protection against queue backlog: when workers were down for an hour, you do not want 50,000 stale "good morning" jobs to flood out afterwards.

Revoking a task

revoke tells workers to skip a task by id:

Python
result = process_video.delay(video_id)

app.control.revoke(result.id)                    # skip if not started yet
app.control.revoke(result.id, terminate=True)    # also kill if already running
app.control.revoke(result.id, terminate=True, signal="SIGKILL")

How it works: the revoke is broadcast to all workers, which keep a set of revoked ids in memory. A worker that receives a message whose id is in the set drops it. Therefore:

  1. Workers must be running to hear the broadcast. Start them with --statedb=/var/run/celery/worker.state so the revoked list survives restarts.
  2. With terminate=True the child process is killed with a signal; this is dangerous (no cleanup) and may kill an unrelated task if the process had already moved on. Use it sparingly.
  3. Revoking a task that has already finished does nothing.
  4. You can revoke by header too: app.control.revoke_by_stamped_headers({"user": "42"}) for tasks you stamped on creation.

For long-running cooperative work, a safer design is cancel flags: store cancelled:<job_id> in Redis or the database and have the task check it between steps, then stop gracefully.

Python Stand-in: a worker with an in-memory revoked set and expiry

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Putting it together

A robust long task combines every tool in this lesson:

Python
@app.task(
    bind=True,
    soft_time_limit=280,
    time_limit=300,
    acks_late=True,
    autoretry_for=(ConnectionError,),
    retry_backoff=True,
    max_retries=3,
)
def export_data(self, job_id):
    for page in pages(job_id):
        if cancelled(job_id):                 # cooperative cancellation
            return "cancelled"
        write_page(page)

and the caller adds expires=3600 so a job that nobody started within an hour is dropped.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the difference between a soft and a hard time limit?

    The soft limit is catchable; the hard limit terminates the process with no chance to react.
  2. 2. What does expires=600 do?

    Expiry is checked on receipt and drops stale messages; it does not interrupt a running task.
  3. 3. Why should workers be started with --statedb when you use revoke?

    Without a state database the revoked list is lost when the worker restarts, and revoked tasks could run.
  4. 4. Which is a safer way to cancel a long running task than revoke(terminate=True)?

    Cooperative cancellation lets the task clean up and avoids killing a child process that may be doing something else.

Summary

  • Use soft_time_limit (catch SoftTimeLimitExceeded) below a hard time_limit so tasks can clean up before being killed.
  • expires drops stale messages when they are received; it does not stop running tasks.
  • revoke is a broadcast stored in worker memory; use --statedb and avoid terminate=True unless necessary.
  • Prefer cooperative cancellation flags for long work, and always set timeouts on network and database calls.

Next: Failure handling and callbacks