Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 36 ยท Scheduling and Beat

Dynamic Schedules: Database-Backed Periodic Tasks

Change Celery Beat schedules at runtime without restarting: custom schedulers, django-celery-beat, SQLAlchemy and Redis schedulers, and how Beat detects changes.

A schedule written in beat_schedule is part of your code. To change it you edit the file, deploy, and restart Beat. That is perfectly fine for system jobs such as "clean up sessions every hour", but not for user-managed schedules: a customer who picks "send me this report every Friday at 17:00", an admin who wants to pause a job for a weekend, or a per-tenant sync with different intervals.

For those cases you store schedule entries in a database and use a scheduler that reads from it.

The scheduler is pluggable

Beat's default scheduler class is celery.beat.PersistentScheduler (it keeps last_run in a local shelve file). You can select another one:

Terminal
celery -A shop beat --scheduler django_celery_beat.schedulers:DatabaseScheduler

or in settings:

Python
app.conf.beat_scheduler = "django_celery_beat.schedulers:DatabaseScheduler"

A scheduler has to answer a few questions: which entries exist, when is each due, and when should Beat look again? Beat calls tick() repeatedly; the scheduler returns how many seconds Beat may sleep. A database scheduler also needs to notice that rows changed.

django-celery-beat

For Django projects the usual choice is the django-celery-beat package.

Terminal
pip install django-celery-beat
Python
# settings.py
INSTALLED_APPS = [
    # ...
    "django_celery_beat",
]
CELERY_BEAT_SCHEDULER = "django_celery_beat.schedulers:DatabaseScheduler"
Terminal
python manage.py migrate
celery -A shop beat -l INFO

The package adds models, all visible in the Django admin:

Model Purpose
PeriodicTask One scheduled entry: task name, args, kwargs, queue, enabled flag, expiry, last run.
IntervalSchedule "Every N seconds/minutes/hours/days".
CrontabSchedule Cron fields plus a timezone.
SolarSchedule Sun events for a location.
ClockedSchedule Run once at a specific time.
PeriodicTasks A single row whose last_update Beat watches for changes.

Create entries from code:

Python
import json
from django_celery_beat.models import CrontabSchedule, PeriodicTask

schedule, _ = CrontabSchedule.objects.get_or_create(
    minute="0", hour="17", day_of_week="5",
    day_of_month="*", month_of_year="*", timezone="Europe/Paris",
)

PeriodicTask.objects.update_or_create(
    name=f"weekly-report-user-{user.id}",
    defaults={
        "task": "reports.tasks.send_weekly_report",
        "crontab": schedule,
        "args": json.dumps([user.id]),
        "enabled": True,
    },
)

Everything in the admin works too: tick Enabled off to pause a job, edit the interval, add a one-off clocked task. No restart is needed. When you save a PeriodicTask, a signal updates the PeriodicTasks.last_update row; the scheduler checks that timestamp on each tick and reloads the schedule if it changed.

Two practical notes:

  • Use get_or_create for schedule rows. They are shared, so a hundred tasks every five minutes can point at one IntervalSchedule.
  • Pass args and kwargs as JSON strings.

How change detection works

A database scheduler does not hold the entries in code, so it must notice edits. The simplest method, used by django-celery-beat, is a version stamp: every write bumps a single "last updated" value, and each Beat tick compares it with the one it loaded. This stand-in implements the idea: a fake database, a scheduler that reloads when the stamp moves, and an operator who edits entries while Beat runs. Time is simulated.

Python Stand-in: a database scheduler that notices changes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The new job starts firing without restarting anything, and the paused job stops. The only cost is the version check on every tick, which is one tiny query.

Other dynamic schedulers

  • celery-sqlalchemy-scheduler brings the same idea to non-Django projects: tables for periodic tasks in any SQLAlchemy database.
  • redbeat stores schedule entries in Redis and uses a lock key so that only one Beat instance is active, which helps high-availability setups (see Beat in production).
  • Custom scheduler: subclass celery.beat.Scheduler and implement setup_schedule, schedule (property) and sync. This is the way to read entries from your own table:
Python
from celery.beat import Scheduler, ScheduleEntry
from celery.schedules import schedule as interval

class MyScheduler(Scheduler):
    max_interval = 30             # re-check at least every 30 seconds

    def setup_schedule(self):
        self.install_default_entries(self.schedule)
        self.update_from_dict(load_entries_from_my_table())

    def tick(self, *args, **kwargs):
        refresh_from_my_table_if_changed(self)       # your change detection
        return super().tick(*args, **kwargs)

Most teams never need to write one; use an existing package.

Entries added on the fly with beat_schedule

You can modify app.conf.beat_schedule in a running Beat from code inside Beat's own process, but the default scheduler does not re-read it. If you need dynamic behaviour, use a database scheduler instead of trying to mutate the dict.

One-off scheduled work

A common request is "run this once at 17:00 on Friday". You have two options:

  1. task.apply_async(eta=datetime(...)): the message sits in a worker's memory until the time arrives. Simple, but lost if the worker restarts without the broker holding it (with Redis, watch the visibility timeout), and not visible or editable.
  2. A clocked schedule row in django-celery-beat with one_off=True: stored in the database, visible in the admin, survives restarts, and disables itself after running.

For long-term or user-visible schedules, prefer a database entry. For delays of seconds or minutes, countdown is fine.

Designing user-facing schedules

  • Store the schedule, not a task per execution. One row like "user 42, every Friday 17:00" is better than creating future tasks.
  • Validate user input into a bounded set of options (daily, weekly, monthly) rather than raw cron strings, or run cron input through a validator and enforce a minimum interval so that a user cannot schedule a job every second.
  • Store the timezone with the schedule and show next run times in the interface.
  • Guard against abuse and growth: cap schedules per user, and make sure the periodic task is cheap, because ten thousand schedules at "every Friday 17:00" means ten thousand messages in the same minute. Spread the load with a dispatcher task: one entry fires every minute, finds the schedules due now in the database and enqueues the real work in small batches.

That last pattern is worth seeing. This runnable stand-in implements the dispatcher: one tick scans due rows and enqueues work in chunks.

Python Stand-in: a dispatcher task that fans out due schedules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When should you use a database-backed scheduler instead of beat_schedule?

    beat_schedule lives in code and needs a restart. A database scheduler reads entries that users or admins can change at runtime.
  2. 2. How does django-celery-beat notice that a PeriodicTask was edited?

    Saving a task bumps the PeriodicTasks.last_update value; the scheduler reloads when it differs from the loaded version.
  3. 3. What is a good approach for 10,000 users who each want a report every Friday at 17:00?

    A dispatcher keeps the schedule in your data, spreads the load and keeps Beat's own list small.
  4. 4. Which is a drawback of apply_async(eta=...) for a job scheduled next week?

    Far-future ETAs are invisible in your own data and interact with broker settings such as the visibility timeout.

Summary

  • The default schedule is code; use a database scheduler (django-celery-beat, SQLAlchemy or RedBeat) for schedules that change at runtime.
  • Database schedulers reload when a version stamp changes, so no restart is needed.
  • Use clocked rows for one-off future jobs and avoid far-future ETAs.
  • For many user schedules, store one row per schedule and run a dispatcher task that fans out the due ones in batches.

Next: Running Beat in Production: Single Instance, Locks and Overlap