Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 36 ยท Scheduling and Beat

Periodic Tasks with Celery Beat

Run tasks on a schedule with Celery Beat: beat_schedule, interval schedules, add_periodic_task, starting beat, the schedule file, and how Beat and workers fit together.

Many applications need work that happens on a clock: send a digest every morning, delete expired sessions every hour, refresh exchange rates every five minutes. Celery's scheduler, Beat, covers this. Beat is a small process that reads a schedule, and each time an entry is due it publishes an ordinary task message to the broker. Workers do the actual work.

flowchart LR
    S["beat_schedule<br/>(entries)"] --> B["celery beat"]
    B -->|"publishes when due"| Q["broker queue"]
    Q --> W["worker"]

Beat never runs your function. It only sends the message at the right moment. No worker, no execution.

Your first schedule

The schedule is the beat_schedule setting: a dictionary of named entries.

Python
from celery import Celery

app = Celery("shop", broker="redis://localhost:6379/0")

app.conf.beat_schedule = {
    "refresh-rates-every-5-minutes": {
        "task": "shop.tasks.refresh_rates",
        "schedule": 300.0,                    # seconds
    },
    "say-hello-every-10-seconds": {
        "task": "shop.tasks.say_hello",
        "schedule": 10.0,
        "args": ("world",),
        "kwargs": {"loud": True},
        "options": {"queue": "default", "expires": 8},
    },
}

Each entry has:

Key Meaning
task The task name (a string, as registered by the worker).
schedule How often: seconds as a number, a timedelta, a crontab, or a solar schedule.
args, kwargs Arguments for the task.
options Any apply_async option (queue, expires, priority...).
relative For timedelta schedules: round to the start of the unit.

The entry's name (the dictionary key) identifies it in the schedule file; keep it stable and descriptive.

Starting Beat

Terminal
celery -A shop beat --loglevel=INFO

You also need a worker, in another terminal:

Terminal
celery -A shop worker --loglevel=INFO

For development you can run both in one process (not for production):

Terminal
celery -A shop worker --beat --loglevel=INFO

Beat writes a small file named celerybeat-schedule in the working directory. It remembers when each entry last ran, so after a restart Beat knows what is due and does not fire everything at once or lose its place. Add it to .gitignore. If you want a different location:

Terminal
celery -A shop beat -s /var/run/celery/celerybeat-schedule

How Beat decides what is due

Beat is a loop: look at each entry, compute when it should next run, sleep until the earliest one (but at most beat_max_loop_interval, 5 minutes by default), publish what is due, repeat. For a plain interval schedule, "due" means at least N seconds since the last run. A key detail: the interval is measured from when the entry last ran, or from Beat's start if it never ran, not from a wall clock boundary. 300.0 does not mean "at :00, :05, :10", it means "every 300 seconds after the last run". Use crontab (next lesson) when you want clock-aligned times.

The stand-in below implements this loop on a simulated clock, so you can watch a schedule with two entries, a restart and a persisted "last run".

Python Stand-in: a Beat loop with interval entries

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the times: say_hello fires at 10, 20, 30; after the restart it fires at 40 because more than ten seconds had passed since its last run. If Beat had forgotten its state (no schedule file), every entry would restart counting from the new start time instead.

Adding entries in code

Instead of a big dictionary you can register entries as the app is configured, using the on_after_configure signal:

Python
@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    sender.add_periodic_task(30.0, ping.s("ok"), name="ping every 30 seconds")
    sender.add_periodic_task(
        crontab(hour=7, minute=30, day_of_week=1),
        weekly_report.s(),
        name="monday report",
    )

add_periodic_task takes a schedule and a signature (task.s(...)), which keeps task references typed rather than strings. Entries added this way and entries in beat_schedule end up in the same list.

Timezone

Beat uses the app's timezone setting for its calculations:

Python
app.conf.timezone = "Europe/Berlin"
app.conf.enable_utc = True

With enable_utc=True (the default) messages carry UTC times, and crontab entries are interpreted in the configured timezone. When you change either setting, restart Beat and delete the schedule file, because stored last-run times use the old setting. Daylight-saving changes can make a local 02:30 run twice or not at all in some zones; for critical jobs prefer UTC.

Making periodic tasks robust

A scheduled task is still a normal task, and the usual rules apply with extra weight:

  • Idempotent. If Beat is restarted or two Beat instances accidentally run, the task may be sent twice.
  • expires. A task scheduled every minute that waits behind a backlog is pointless once the next one is due. Set expires slightly below the interval: "options": {"expires": 55}.
  • Fast or chunked. A task that takes longer than its interval overlaps itself. Make it quicker, or add a lock (see Beat in production).
  • Do not use Beat for precise timing. Typical accuracy is within a second or two; the message may then wait in a queue. For one-off delayed jobs, use apply_async(eta=...) instead of Beat.

Troubleshooting

Symptom Likely cause
Beat logs "Sending due task" but nothing runs No worker is consuming the queue, or the task name does not match a registered task.
Nothing is published The entry's schedule has not elapsed yet, or the beat process died.
Tasks fire twice Two Beat instances are running.
A change to the schedule has no effect Beat must be restarted (the default scheduler reads beat_schedule at start).
Times are off by hours timezone mismatch; check enable_utc and delete the schedule file.
KeyError / "unregistered task" in the worker The entry's task name is wrong.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which process actually executes a periodic task's function?

    Beat only publishes the message when the entry is due. Workers run the function.
  2. 2. What does "schedule": 300.0 mean?

    A numeric schedule is an interval, not a clock time. Use crontab for clock-aligned runs.
  3. 3. What is the purpose of the celerybeat-schedule file?

    The default persistent scheduler stores last-run times in this file.
  4. 4. Why set expires slightly below the interval for a frequent task?

    If the queue is backed up, an old run is worthless once the next one is due, so it should expire.

Summary

  • Beat reads beat_schedule (or add_periodic_task calls) and publishes task messages when entries are due; workers execute them.
  • Numeric schedules are intervals measured from the last run; the schedule file keeps last-run times across restarts.
  • Run exactly one Beat process, restart it after schedule changes, and set the timezone deliberately.
  • Make periodic tasks idempotent, give them expires, and keep them shorter than their interval.

Next: Crontab and Solar Schedules