Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 36 ยท Scheduling and Beat

Running Beat in Production: Single Instance, Locks and Overlap

Run Celery Beat safely: exactly one instance, supervision, lock-based overlap prevention, catching up after downtime, monitoring missed runs and high-availability options.

Beat looks simple in a tutorial. In production it has two awkward properties: you must run exactly one of it, and it gives no guarantee about missed or overlapping runs. This lesson covers the habits that make scheduled work dependable.

Rule 1: exactly one Beat

If you start two Beat processes with the same schedule, every entry is published twice. If you start none (because a deploy forgot it), nothing is published and nobody notices until the data is stale.

Typical ways to guarantee one instance:

  • A dedicated process or container with replicas: 1. In Kubernetes, a Deployment with one replica and strategy: Recreate (so the old pod stops before the new one starts).
  • Systemd or Supervisor on one machine, with autorestart enabled.
  • Do not embed Beat in every worker. The -B flag (celery worker -B) is for development; with ten workers you would get ten Beats.
  • Careful with autoscaled web servers: never start Beat from the web process.

A minimal systemd unit:

INI
[Unit]
Description=Celery Beat
After=network.target

[Service]
User=app
WorkingDirectory=/srv/shop
ExecStart=/srv/shop/.venv/bin/celery -A shop beat -l INFO --pidfile=/run/celery/beat.pid -s /var/lib/celery/beat-schedule
Restart=always
RuntimeDirectory=celery
StateDirectory=celery

[Install]
WantedBy=multi-user.target

Store the schedule file on durable storage (StateDirectory), or use a database scheduler, so restarts keep last-run times.

Rule 2: tolerate duplicates and overlaps

Even with discipline, a task can be published twice (a failover, an accidental second Beat) or run while a previous run is still going (a job that takes 7 minutes on a 5-minute schedule). Your options:

  1. Make it idempotent, so a second run is harmless.
  2. Make it quick, or split work so one run does little.
  3. Add a lock so overlapping runs exit immediately.

A lock is easy to build on Redis or the database. The key is a lock with an expiry, so that a crashed worker cannot hold it forever:

Python
import redis
from celery import shared_task

r = redis.Redis()

@shared_task(bind=True)
def sync_inventory(self):
    # SET key value NX EX 300: only succeeds if the key does not exist
    got_lock = r.set("lock:sync_inventory", self.request.id, nx=True, ex=300)
    if not got_lock:
        return "skipped: previous run still active"
    try:
        do_sync()
    finally:
        # release only if we still own the lock
        if r.get("lock:sync_inventory") == self.request.id.encode():
            r.delete("lock:sync_inventory")

(The check-then-delete in finally is not perfectly atomic; production code uses a Lua script or a library such as redis-py's Lock. The expiry is the real safety net.)

Here is the same logic as a runnable stand-in with a lock table, a simulated clock, and a task that takes longer than its schedule interval.

Python Stand-in: a lock with expiry prevents overlapping runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without the lock, runs 2 and 3 would execute in parallel with earlier ones. With it, they stand aside and a later tick finds the lock free. If a worker had crashed, the lock would simply expire after ttl.

Rule 3: design for missed runs

Beat gives at-most-once publishing per schedule slot, and if Beat or all workers are down, the slot is lost. Handle it by writing tasks as "bring the state up to date", not "do this slot's work":

  • Instead of send_digest(date=today), write send_pending_digests() which finds every user whose digest is overdue.
  • Instead of "import yesterday's file", loop over all dates not yet imported.
  • Keep a last_successful_run timestamp in your database; each run processes (last_run, now].

Here is that pattern as a runnable stand-in. Three runs are missed, and the next run repairs the gap.

Python Stand-in: catch-up tasks repair missed runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rule 4: monitor the schedule itself

"Beat is running" and "the work happened" are different facts. Ways to watch:

  • Heartbeat task: a periodic task that writes the current time to Redis or a database; alert if the value is older than a few minutes.
  • Dead man's switch: services such as healthchecks-style endpoints expect a ping at each run; no ping means an alert.
  • Output checks: alert when a table has no rows from today, or a queue length rises while consumers stay at zero.
  • Logs: Beat logs Scheduler: Sending due task <name>; count them.

Also watch the queue length of the queue Beat publishes to. If workers are stuck, periodic tasks pile up and, when the workers recover, run in a burst. This is another reason to set expires on frequent entries.

High availability for Beat

If a single Beat is a risk you cannot accept, use a scheduler that supports leader election: redbeat keeps schedule entries in Redis and uses a lock with a timeout so that one of several Beat processes is active and others take over after a failure. Orchestrators can also restart a failed Beat in seconds, which is often good enough. Pairing a database scheduler with an orchestrator and catch-up tasks usually gives a better return than building leader election yourself.

A deployment checklist

  • One Beat instance, supervised, with a restart policy and a stop-before-start rollout.
  • Schedule state on durable storage (or a database scheduler).
  • Timezone set deliberately, preferably UTC; Beat restarted after any timezone change.
  • Tasks idempotent, quick, with expires and, if needed, a lock.
  • Catch-up logic for work that must not be skipped.
  • Alerts on staleness of the output, not only on process liveness.
  • Workers started for the queues Beat publishes to, and the new code deployed to both Beat and workers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens if two Beat instances run against the same schedule?

    The default scheduler has no leader election, so each instance publishes the due tasks.
  2. 2. Why does a lock for overlapping runs need an expiry?

    If the owner dies before releasing, the lock must time out so later runs can proceed.
  3. 3. Beat was down for two hours. How should a daily-import task be written so nothing is lost?

    A catch-up task processes all outstanding work, so a missed slot is repaired on the next run.
  4. 4. Which is the best way to detect that periodic work has silently stopped?

    Process liveness does not prove the work happened. Watch the result.

Summary

  • Run exactly one supervised Beat; never use -B in production workers.
  • Expect duplicates and overlap: idempotent tasks, expires, and Redis or database locks with an expiry.
  • Beat does not replay missed slots, so write catch-up tasks that process everything outstanding.
  • Monitor the output and a heartbeat, and consider RedBeat or an orchestrator for failover.

Next: Celery Signatures: Partials, Immutability and Options