Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 36 ยท Tasks and Workers

Defining Tasks with @app.task and Task Options

Learn how to define Celery tasks: the decorator, task names, bind=True and self.request, per-task options, custom base classes and the names that workers use.

A task is the unit of work in Celery. Technically it is a class instance created from your function by a decorator, but you almost never need to think about the class. What matters is what the decorator records: a name that identifies the task in messages, the function to run, and a set of options that control how it behaves.

The basic decorator

Python
from celery import Celery

app = Celery("shop", broker="redis://localhost:6379/0")

@app.task
def send_receipt(order_id):
    ...

After this, send_receipt is a task object. It is still callable, so send_receipt(5) runs the function in the current process. It also gains methods such as delay(), apply_async(), s() (signatures) and retry(), which you will meet in later lessons.

You can also give options to the decorator, with or without parentheses:

Python
@app.task(name="shop.receipts.send", ignore_result=True, rate_limit="10/m")
def send_receipt(order_id):
    ...

Task names

Every message carries the task name as a string. The worker uses it to find the function in its registry. If you do not set name=, Celery builds one from the module and function: shop.tasks.send_receipt.

This is the source of a classic problem. The name depends on the import path. If one process imports the module as tasks and another as myproject.tasks, the producer sends tasks.add while the worker only knows myproject.tasks.add, and you get Received unregistered task of type 'tasks.add'.

Two ways to avoid it:

  1. Always import tasks through the same full package path.
  2. Give important tasks an explicit name: @app.task(name="billing.charge"). An explicit name survives refactors such as moving a function to another file, and lets old queued messages still find the task after a deploy.

This stand-in shows how automatic names depend on where the function lives, and how an explicit name stays stable:

Python Stand-in: automatic versus explicit task names

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

bind=True and self

By default the function receives only your arguments. With bind=True the first argument becomes the task instance itself, conventionally named self. That gives access to the current request (id, retries, headers) and to self.retry():

Python
@app.task(bind=True)
def import_file(self, path):
    print("task id:", self.request.id)
    print("attempt:", self.request.retries)
    print("delivery info:", self.request.delivery_info)
    ...

Useful fields of self.request:

Field Meaning
id The unique task id (UUID string).
retries How many times this task has been retried so far.
args, kwargs The arguments it was called with.
eta, expires Timing options, if set.
hostname The worker that runs it.
delivery_info Queue and routing key it arrived on.
is_eager True when run locally without a worker.
headers, parent_id, root_id Workflow and tracing information.

The stand-in below simulates a bound task and its request context, including how retry counting works. Real retries are covered in a later chapter; here the point is that self is the doorway to the context.

Python Stand-in: bind=True and self.request

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Per-task options

Options passed to @app.task(...) override app-wide settings for that one task. The ones you will use most:

Option What it does
name Explicit task name.
bind Pass the task instance as the first argument.
ignore_result Do not store the return value. Saves backend traffic for fire-and-forget tasks.
rate_limit Limit how often the task starts on a worker, such as "100/m".
time_limit, soft_time_limit Seconds before the task is killed or gets an exception.
acks_late Acknowledge after the task finishes instead of before.
autoretry_for, retry_backoff, max_retries Declarative retry rules.
queue A default queue for this task (see Queues and routing).
serializer Override the serializer for this task.
base A custom task class.

Rate limits are per worker instance, not cluster-wide, and they use a token bucket: "10/m" allows about one start every six seconds, not ten at once.

Custom task classes

If many tasks share behaviour (logging, error handling, database cleanup), put it in a base class. Celery calls hooks such as on_success, on_failure, on_retry and after_return:

Python
from celery import Task

class LoggedTask(Task):
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        print(f"task {self.name}[{task_id}] failed: {exc!r}")

    def on_success(self, retval, task_id, args, kwargs):
        print(f"task {self.name}[{task_id}] returned {retval!r}")

@app.task(base=LoggedTask, bind=True)
def risky(self, x):
    return 10 / x

One subtlety: by default Celery creates one instance per task, shared by every call in that worker process. Do not store per-call state on self; use local variables. Instance attributes are only safe for things you want shared, such as a lazily created connection:

Python
class DatabaseTask(Task):
    _db = None

    @property
    def db(self):
        if self._db is None:
            self._db = connect()      # created once per worker process
        return self._db

Here is a runnable stand-in of the hook idea. A tiny runner calls on_success or on_failure based on the outcome:

Python Stand-in: success and failure hooks in a base class

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rules for the function body

  • Keep the function small and focused. Do one job; chain jobs with workflows.
  • Make the arguments simple and serializable (see Task design rules).
  • Do not call .get() on another task's result inside a task. Waiting inside a worker can deadlock when all worker slots are busy waiting for each other. Use chains or callbacks instead.
  • Do not rely on global mutable state: with several worker processes, each has its own copy.
  • Log with the standard logging module, using celery.utils.log.get_task_logger(__name__) so lines carry the task name and id.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a worker use to find the function for an incoming message?

    The message carries only the task name; the worker looks it up in its registry of imported tasks.
  2. 2. Why give an important task an explicit name=?

    Automatic names follow the module path, so refactors or inconsistent imports break matching between producer and worker.
  3. 3. What does bind=True give you?

    With bind=True the function receives self, the task instance.
  4. 4. Why should you avoid storing per-call state in instance attributes of a task class?

    The same task object handles every call, so per-call data stored on self would leak between calls.

Summary

  • @app.task registers a function under a name; producers and workers must agree on that name.
  • Use explicit names for tasks that matter; use bind=True to reach self.request and self.retry.
  • Per-task options (ignore_result, rate_limit, time limits, acks_late, retries) override app settings.
  • Share behaviour with a custom base class, but keep per-call state in local variables.

Next: Calling Tasks: delay, apply_async and Options