Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 40 ยท Projects and Practice

Project: A Job Scheduler with Graceful Shutdown

Build a periodic job scheduler with asyncio: per-job loops, overlap protection, failure isolation, graceful shutdown, and a virtual clock that makes the whole run deterministic.

Almost every long-running service needs background jobs that repeat: refresh a cache every minute, flush metrics every ten seconds, send a daily report. In this project you will build a small asyncio job scheduler and, along the way, solve the problems that make such code fragile: jobs that outlive their interval, jobs that crash, and a shutdown that must not abandon work in the middle.

To make a scheduler testable, time itself has to be controllable. So the first thing you will build is a tiny virtual clock: instead of really waiting 60 seconds, a driver jumps from one wake-up time to the next. The same technique is how you test time-dependent async code in general (see testing async code). The virtual seconds in the output cost no real time, and the output is the same on every run.

Step 1: a virtual clock

A VirtualClock has a sleep() coroutine that registers a wake-up time and waits on a future, and a driver run_until(t) that repeatedly: lets every task run until all are idle, finds the earliest wake-up, sets the clock to it and wakes the sleeper.

Python
import asyncio, heapq, itertools

class VirtualClock:
    def __init__(self):
        self.now = 0.0
        self._sleepers = []                 # heap of (wake_time, tiebreaker, future)
        self._counter = itertools.count()

    async def sleep(self, seconds):
        future = asyncio.get_running_loop().create_future()
        heapq.heappush(self._sleepers, (self.now + seconds, next(self._counter), future))
        await future

    async def _settle(self):
        for _ in range(30):                 # let every ready task run until it blocks
            await asyncio.sleep(0)

    async def run_until(self, target):
        while True:
            await self._settle()
            while self._sleepers and self._sleepers[0][2].done():
                heapq.heappop(self._sleepers)            # skip cancelled sleepers
            if not self._sleepers or self._sleepers[0][0] > target:
                self.now = target
                await self._settle()
                return
            when, _, future = heapq.heappop(self._sleepers)
            self.now = when
            future.set_result(None)

It is only about twenty lines, but it replaces asyncio.sleep with something we control completely. In the real scheduler, you pass asyncio.sleep instead and use time.monotonic() for the clock.

Step 2: the design

flowchart TD
    S["Scheduler.start()"] --> L1["loop: heartbeat"]
    S --> L2["loop: report"]
    S --> L3["loop: flaky"]
    L2 -->|"interval elapsed"| C{"previous run still going?"}
    C -->|"yes"| K["skip this run"]
    C -->|"no"| E["run the job in its own task"]
    E -->|"exception"| LOG["log failure, keep scheduling"]

Each job has its own loop. A run is a separate task so the loop keeps its rhythm and the job's failure cannot kill the schedule.

Rules the scheduler enforces:

  1. One loop per job, each sleeping its own interval, so a slow job never delays the others.
  2. Overlap policy: skip. If the previous run is still going when the next is due, skip the new run (and log it) instead of piling up work. Other policies are "queue" and "run in parallel"; choose deliberately.
  3. Failure isolation. An exception in a job is logged and the schedule continues.
  4. Graceful shutdown. Stop starting new runs immediately, but let runs already in progress finish.
  5. No lost tasks. The scheduler keeps a reference to every running execution.

Step 3: the scheduler and a demo run

Python The scheduler on a virtual clock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Study the log. heartbeat fires every ten virtual seconds. flaky fails on every second run, but its loop keeps going: the failure is logged and isolated. report takes 30 seconds against a 25-second interval, so its tick at t=50 is skipped with a log line instead of two reports running at once. The first report run (started at 25) finishes at 55. At the moment of shutdown (t=60) the heartbeat and flaky runs have just started; they are allowed to finish, which is why the log still shows their results at t=61 and why we advance the clock afterwards.

Step 4: real-time wiring

On a real machine only two lines change: construct the scheduler with a real clock object that offers sleep and now:

Python
import time

class RealClock:
    @property
    def now(self):
        return time.monotonic()
    async def sleep(self, seconds):
        await asyncio.sleep(seconds)

and trigger shutdown() from a signal handler, then wait for in-flight runs with a time limit:

Python
async def main():
    scheduler = Scheduler(RealClock())
    scheduler.add("heartbeat", 10, heartbeat)
    scheduler.start()

    stop = asyncio.Event()
    loop = asyncio.get_running_loop()
    for sig in (signal.SIGINT, signal.SIGTERM):
        loop.add_signal_handler(sig, stop.set)
    await stop.wait()

    await scheduler.shutdown()
    in_flight = list(scheduler.running.values())
    if in_flight:
        async with asyncio.timeout(30):                     # grace period
            await asyncio.gather(*in_flight, return_exceptions=True)

(Signal handlers do not exist in the browser, so this snippet is read-only. See shielding and cleanup for the shutdown pattern.)

Design questions to think about

  • Drift. sleep(interval) after each run drifts, because the run time is added to the period. To keep a steady rhythm, compute the next wake-up from a schedule (next_due += interval) and sleep for next_due - now.
  • Overlap policy. We skip. A "queue" policy would count missed runs; a "parallel" policy would start another task each time, which risks the unbounded concurrency described in rate limiting and concurrency limits.
  • Jitter. If a hundred processes all schedule "every minute on the minute", they hit the database together. Add a random offset.
  • Persistence. In-process schedulers forget everything on restart. Persistent, distributed jobs belong in a task queue such as Celery or a cron service; an in-process scheduler is right for caches, metrics and housekeeping.
  • Timeouts per job. Wrap func(self) in asyncio.timeout(job.timeout) so a hung job cannot hold its slot forever (timeouts).

Extensions to try

  1. Add a per-job timeout and log TIMED OUT on TimeoutError.
  2. Add run_now(name) to trigger a job manually, and make it respect the overlap rule.
  3. Add a "queue" overlap policy that runs the missed execution right after the previous one ends.
  4. Replace fixed intervals with a tiny cron expression parser for "every day at 03:00".
  5. Use a TaskGroup to own the loops, so an unexpected error in the scheduler itself stops everything cleanly.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does each job get its own loop instead of one loop that checks all jobs?

    Independent loops mean one job's duration never blocks another job's schedule.
  2. 2. What is the purpose of running each execution as a separate task?

    The loop only starts and tracks executions. The job runs in its own task, whose exceptions are caught and logged without affecting the schedule.
  3. 3. What does a virtual clock make possible?

    Time only advances when the driver says so, so tests run quickly and always give the same result.
  4. 4. On graceful shutdown, what should happen to runs that are already in progress?

    Stop scheduling, then wait for in-flight work with a time limit, so jobs are not abandoned half-done.

Summary

  • A scheduler is a loop per job, a separate task per run, an overlap policy and failure isolation.
  • A virtual clock (a heap of sleepers plus a driver) makes time-based async code fast and deterministic to test.
  • Graceful shutdown stops new runs first, then waits for in-flight ones with a deadline.
  • Watch out for drift, overlap, thundering herds and the fact that in-process schedulers are not persistent.

Next: asyncio practice exercises