Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 36 ยท Workflows: Chains and Groups

Celery Signatures: Partials, Immutability and Options

Learn Celery signatures, the building block of workflows: task.s(), partial arguments, immutable signatures with .si(), options and how arguments are merged.

So far a task has been something you call with .delay(). That is fine for a single job, but real work often looks like "download the file, then resize it, then email the result" or "process 500 rows in parallel and then summarize". To describe such work Celery gives you the canvas: a small set of primitives (signature, chain, group, chord, map, starmap, chunks) that you combine like Lego. Everything in the canvas starts with one idea: the signature.

A signature is a task call that has been packaged but not sent. It remembers the task, its arguments, and its execution options (queue, countdown, retry policy, and so on). You can pass it around, store it, serialize it, combine it with other signatures, and send it later. Think of it as a function call written down on a card.

Celery itself, a broker and workers do not run on this site. The Celery code is read-only. The runnable examples use a small plain-Python stand-in class that follows the same argument rules, so you can experiment with the behavior. Output is shown only for the stand-ins.

Creating signatures

There are three equivalent ways to create one. Assume this task exists:

Python
from celery import Celery

app = Celery("shop", broker="redis://localhost:6379/0")

@app.task
def add(x, y):
    return x + y
Python
from celery import signature

sig_a = add.signature((2, 2), countdown=10)   # the long form
sig_b = add.s(2, 2)                           # shortcut for signature(args)
sig_c = signature("shop.add", args=(2, 2))    # by task NAME, no import needed

add.s(2, 2) is the form you will use ninety percent of the time. The third form is useful when the producer cannot import the task module (for example another service sends work by name). A signature is just a dictionary-like object, so dict(sig_b) shows the task name, args, kwargs and options.

You run a signature in two ways: call it directly (sig_b() runs add(2, 2) in the current process, like a normal function) or send it to a worker with sig_b.delay() / sig_b.apply_async(). Extra options can be added at send time: sig_b.apply_async(countdown=5, queue="math").

Partial signatures

A signature does not need all its arguments up front. add.s(2) is a partial: it knows y (well, the second argument) but waits for the rest.

When you send a partial with more arguments, those new arguments are prepended to the stored ones:

Python
partial = add.s(2)        # waiting for the first number
partial.delay(4)          # worker runs add(4, 2) -> 6

Keyword arguments are merged, with the later values winning:

Python
sig = greet.s("Ada", punctuation="!")
sig.delay(punctuation="?")   # greet("Ada", punctuation="?")

This prepend rule is the key to chains. In a chain, the result of the previous task is passed to the next signature as an extra argument, which is prepended. So add.s(2) in the middle of a chain receives add(previous_result, 2). You will see this again in Chains.

Python Stand-in: a signature that merges arguments

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Immutable signatures

Sometimes a step in a chain must not receive the previous result. For example send_report just needs a report id you already know, and passing it an unexpected extra argument would raise TypeError. Make the signature immutable with .si() (or immutable=True):

Python
add.si(2, 2)        # always add(2, 2), ignores any incoming result
add.s(2, 2)         # mutable: in a chain becomes add(prev, 2, 2) -> TypeError

Rule of thumb: use .s() when the step wants the previous result, and .si() when it is an independent step that only needs to run after the previous one.

Options on a signature

Options set on the signature travel with it. These are the same options apply_async accepts:

Python
sig = add.s(4, 4).set(queue="math", countdown=30, expires=300)

.set(...) returns the signature, so calls can be chained fluently. Useful options are queue, countdown, eta, expires, priority, retry, link (callback on success) and link_error (callback on failure). Options given to apply_async at send time override the stored ones.

Python Stand-in: immutable vs mutable steps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other handy signature methods

  • sig.clone(args=(...), kwargs={...}) returns a copy with extra arguments, leaving the original intact.
  • sig.apply() runs it locally in the current process (handy in tests). Setting task_always_eager = True does the same for .delay() everywhere.
  • sig.freeze() assigns a task id in advance, so you can store the id before sending.
  • sig | other is the pipe operator that builds a chain.
  • sig.link(callback) and sig.link_error(errback) attach success/failure callbacks (see Failure handling).

Signatures are serializable: dict(sig) is JSON-friendly, and Signature.from_dict(d) rebuilds one. That is how Celery sends a whole workflow inside a single message.

Common mistakes

  1. Calling the task instead of making a signature: add(2, 2) runs it now; add.s(2, 2) only describes it.
  2. Forgetting immutability in a chain, which causes an "unexpected argument" TypeError in the worker log.
  3. Expecting appended arguments. New positional arguments go in front, not behind.
  4. Passing a signature object as a task argument where a plain id would do. Keep message payloads small.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does add.s(2, 2) do?

    A signature packages a task call so you can send or combine it later. Nothing is sent until you call delay, apply_async or put it in a workflow.
  2. 2. partial = add.s(2); partial.delay(4) makes the worker run which call?

    Arguments supplied later are prepended to the stored ones, so the result is add(4, 2).
  3. 3. When do you use .si() instead of .s() in a chain?

    An immutable signature never receives the parent's return value; it only runs after the parent finished.
  4. 4. Which method lets you attach a queue or countdown to a signature?

    set() stores options on the signature and returns it, so you can keep chaining calls.

Summary

  • A signature is a task call packaged with its arguments and options but not yet sent.
  • task.s(...) creates it; task.si(...) creates an immutable one; signature("name") works without importing the task.
  • Partial signatures prepend later positional arguments and merge keyword arguments.
  • Use .set() for options such as queue, countdown, expires, link and link_error.
  • Signatures are serializable, which is how whole workflows travel in one message.

Next: Chains: running tasks in sequence