Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 40 ยท Timeouts and Cancellation

Shielding Work and Safe Cleanup in asyncio

Protect critical async work with asyncio.shield, clean up safely after cancellation, understand uncancel and TaskGroup, and shut down background tasks gracefully.

Sometimes you must not be interrupted halfway. Imagine saving a file, committing a database transaction or releasing a lock: if cancellation lands in the middle, you may leave a mess behind. asyncio offers asyncio.shield() for work that should survive a cancel, and a handful of patterns for cleanup that must run even though the task is being cancelled.

This lesson builds on cancelling tasks. You will learn what shield() really protects (less than people assume), how to write cleanup code that is itself cancellation-safe, how uncancel() fits in, and how to shut a program with background tasks down gracefully.

asyncio.shield()

await asyncio.shield(aw) runs aw as an inner task and returns an outer future. If the outer waiter is cancelled, the cancellation stops at the shield: the inner task keeps running. The caller still receives CancelledError (or TimeoutError when a timeout cancels it), but the protected work continues in the background.

Python Shielded work survives a timeout

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the two differences. With the shield, "report saved" is printed after the caller has given up, and you can still await inner to get the result. Without it, the work is cancelled before it finishes.

What shield does not do

  • It does not protect against the whole program ending. If asyncio.run() finishes (or the loop is stopped), pending tasks are cancelled regardless of the shield.
  • It does not stop the caller from being cancelled. Only the inner task is protected.
  • Keep a reference to the inner task. The loop only keeps weak references to tasks, so an unreferenced shielded task can be garbage collected mid-flight. Creating it with create_task() into a variable or a set, as above, avoids that.
  • If the shielded work raises and nobody awaits the inner task, you get a "Task exception was never retrieved" message. Always have someone await the inner task eventually.

Cleanup that must await

Cleanup code often needs await (close a connection, flush a buffer). But in a finally block of a task that was cancelled, a second cancellation could arrive while you are cleaning up. Two practical rules:

  1. Keep async cleanup short and give it its own timeout.
  2. If it must not be interrupted, run it as a shielded task.
Python Cleanup that finishes even when cancelled

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The impatient second cancel arrives while the worker is inside its cleanup. Thanks to the shield and the inner await cleanup, the connection is still closed before the worker finishes. Most programs do not need this level of care, but libraries that guard external resources often do.

Context managers do this for you

The best way to get safe cleanup is async with. An async with block calls __aexit__ even for cancellation, so resources (locks, connections, files) release themselves. Write your own with contextlib.asynccontextmanager:

Python
from contextlib import asynccontextmanager

@asynccontextmanager
async def connection(pool):
    conn = await pool.acquire()
    try:
        yield conn
    finally:
        await pool.release(conn)

Locks, semaphores and TaskGroups all follow this pattern, so a cancelled task never leaves a lock held.

uncancel() and swallowing cancellation on purpose

Occasionally you are the one who wants to absorb a cancellation, for instance a loop that handles a "stop this one step" cancel but should carry on afterwards. The correct way is to call task.uncancel() after catching it, which reduces the counter that asyncio.timeout() and TaskGroup rely on:

Python
try:
    await step()
except asyncio.CancelledError:
    if asyncio.current_task().uncancel() > 0:
        raise            # someone else (e.g. an outer timeout) also cancelled us
    # otherwise: this cancel was ours to absorb, continue

Most application code should not do this: let cancellation propagate. uncancel() exists mainly for the implementation of timeout() and TaskGroup, which is why those two are correct when nested while hand-rolled equivalents are not. If you ever find yourself writing uncancel(), consider whether a timeout() or TaskGroup expresses it better.

Graceful shutdown of background tasks

A long-running program usually owns several background tasks: a heartbeat, a queue consumer, a scheduler. At shutdown you want to stop them in order and wait for them. The pattern is cancel, then gather:

Python Cancel, then gather

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

return_exceptions=True is what makes gather wait for every task, even though they all end in CancelledError. On a real server you trigger this from a signal handler:

Python
loop = asyncio.get_running_loop()
stop = asyncio.Event()
for sig in (signal.SIGINT, signal.SIGTERM):
    loop.add_signal_handler(sig, stop.set)
await stop.wait()
# then cancel and gather as above

(Signal handlers are not available in the browser, so this snippet is read-only here.) When you can, prefer a TaskGroup for the background tasks: leaving the async with block cancels and awaits all of them in one step.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens to the inner task when the code awaiting asyncio.shield(inner) is cancelled?

    shield() stops the cancellation at the outer future. The caller sees CancelledError, but the inner task continues.
  2. 2. Why should you keep a reference to a shielded inner task?

    The loop holds only weak references to tasks, and you also need the reference to collect the result or exception later.
  3. 3. Which is the best way to ensure a lock is released when a task is cancelled?

    async with calls the exit method on cancellation too, so the lock is always released.
  4. 4. A graceful shutdown cancels several background tasks. Which line ensures all of them have finished before moving on?

    Cancelling only requests. Gathering with return_exceptions=True waits for all tasks to finish their cleanup without raising.

Summary

  • asyncio.shield() lets the inner task continue when its waiter is cancelled; keep a reference and await it later.
  • Put cleanup in finally or async with, keep it short, and shield it if it must not be interrupted.
  • uncancel() is for the rare case of deliberately absorbing a cancellation; normally let it propagate.
  • Shut down by cancelling background tasks and gathering them with return_exceptions=True, or use a TaskGroup.

Next: Retries and exponential backoff