Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 40 ยท Synchronization and Queues

asyncio.Queue and the Producer-Consumer Pattern

Learn asyncio.Queue for passing work between tasks: put, get, maxsize backpressure, task_done and join, graceful stopping with shutdown, and a worker pool example.

Sooner or later two parts of a program need to hand items to each other: a crawler finds links that a downloader must fetch, a web server accepts requests that workers must process, a sensor produces readings that a logger must save. The tool for this is asyncio.Queue, a first-in, first-out buffer designed for tasks. One side puts items in, the other side gets them out, and each side waits politely when the queue is full or empty, without blocking the loop.

This is the producer-consumer pattern, and it is the backbone of many real asyncio programs.

put and get

await queue.put(item) appends an item (waiting if the queue is full). await queue.get() removes and returns the oldest item (waiting if the queue is empty).

Python A producer and a consumer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The consumer starts first and immediately waits because the queue is empty. Each put wakes it up. The special value None is a sentinel that tells the consumer to stop. The producer and consumer never talk to each other directly; they only share the queue, which makes both easy to change and test.

flowchart LR
    P1["Producer"] -->|"put"| Q["Queue<br/>FIFO buffer"]
    Q -->|"get"| C1["Consumer"]

The queue decouples the speed of producers and consumers: each waits only when the buffer is empty or full.

Backpressure with maxsize

By default a queue can grow without limit. If producers are faster than consumers, memory fills up. Setting maxsize makes put wait when the queue is full. That automatically slows the producer down to the consumer's pace, which is called backpressure:

Python maxsize slows the producer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The producer fills the two slots instantly, then must wait for the slow consumer to free space. The queue never holds more than two items. Useful methods: qsize(), empty(), full(), and the non-waiting variants put_nowait(item) (raises QueueFull) and get_nowait() (raises QueueEmpty).

Many workers, task_done and join

Real systems use several consumers ("workers"). Sentinels get awkward then, so asyncio provides a cleaner pair: queue.task_done() (a consumer says "I finished one item") and await queue.join() (blocks until every item ever put has been marked done).

Python Worker pool with join

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pattern has four steps:

  1. Put all the work in the queue.
  2. Start N workers that loop get / work / task_done.
  3. await queue.join() to wait until everything is processed.
  4. Cancel the workers (they would otherwise wait forever on an empty queue).

Three workers share six jobs, and fast workers automatically take more jobs than slow ones: the queue balances the load with no extra code. It is also a clean way to limit concurrency to N workers without creating a task per item.

Forgetting task_done() makes join() wait forever. Put task_done() in a finally block so a failing job cannot leave the queue unfinished.

Stopping workers: shutdown (Python 3.13+)

Cancelling workers works, but a more polite way exists in recent Python: queue.shutdown(). After it, put raises asyncio.QueueShutDown, and get keeps returning remaining items and then raises QueueShutDown, which lets workers exit on their own:

Python queue.shutdown()

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

shutdown(immediate=True) discards the remaining items and wakes everyone at once. On older Python versions use sentinel values or cancellation.

Multiple producers and consumers

Queues work with any number of producers and consumers on either side. For example, a crawler pattern: workers take URLs from a queue, fetch pages, and put newly found URLs back into the same queue. Because join() counts unfinished items, it returns exactly when no work remains and nothing new was added.

Python Workers that add more work

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The seen set prevents visiting a page twice, so the crawl terminates even though pages link back to each other. This structure, scaled up with real HTTP requests, is the skeleton of a web crawler.

Queue versus other tools

  • Queue: items flow from some tasks to others and must not be lost.
  • Event: a yes/no signal with no data.
  • Lock: protects shared state; items do not move.
  • asyncio.PriorityQueue / LifoQueue: same API with different ordering, see the next lesson.

asyncio.Queue is not thread-safe. To feed it from another thread use loop.call_soon_threadsafe(queue.put_nowait, item). It is also not for multiprocessing; use multiprocessing.Queue there.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does await queue.get() do when the queue is empty?

    get() suspends the calling task only; use get_nowait() if you want an exception instead.
  2. 2. What is the purpose of Queue(maxsize=N)?

    A bounded queue slows fast producers down to the speed of consumers and caps memory.
  3. 3. When does await queue.join() return?

    join counts unfinished items: put increments, task_done decrements. An empty queue can still have items in progress.
  4. 4. Why should task_done() be called in a finally block?

    If an error skips task_done(), the unfinished counter never reaches zero.

Summary

  • asyncio.Queue passes items between tasks with put and get, waiting when full or empty.
  • maxsize gives backpressure; put_nowait and get_nowait never wait.
  • Workers use get / work / task_done; join() waits until all items are done.
  • Stop workers by cancelling them, with queue.shutdown() (3.13+), or with sentinels.
  • A worker pool over a queue is also a simple, memory-friendly concurrency limiter.

Next: Priority and LIFO queues