Python Reference ยท Standard Library: Concurrency, Network and Tools
The multiprocessing module
Reference for Python's multiprocessing module: Process, Pool.map, Queue, Value, Manager, start methods, pickling limits and the __main__ guard.
The multiprocessing module runs Python code in separate operating-system processes. Each process has its own interpreter and its own memory, so the global interpreter lock (GIL) is not a bottleneck: CPU-bound work can use every core. The price is that processes do not share ordinary objects. Arguments and results are copied between them by pickling, and shared state needs explicit tools such as queues, pipes, shared values or a manager.
For most jobs you do not create processes by hand. You build a Pool (or concurrent.futures.ProcessPoolExecutor) and map a function over your data; the pool splits the work, runs it in worker processes and returns the results in order.
from multiprocessing import Pool def square(n): return n * n if __name__ == "__main__": with Pool() as pool: print(pool.map(square, range(5))) # [0, 1, 4, 9, 16]
The in-browser runner cannot start processes (the module's C helper _multiprocessing is not available), so Process, Pool, Queue and Manager are shown read-only here. The examples that run below cover what is checkable in a single process: how data must be picklable, and how to structure code so it moves to a pool unchanged.
Try it
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.
Press Run (or Ctrl + Enter) to see the result here.
Program input ยท what you'd type while the program runs
Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.
The same chunks on a pool, as it would run in a normal Python environment (read-only):
from multiprocessing import Pool if __name__ == "__main__": with Pool(processes=4) as pool: results = pool.map(count_primes, chunks(range(2, 20000), 5000)) print(sum(results))
What is in the module
| Name | Description |
|---|---|
Process(target, args, kwargs, daemon) |
One child process. start(), join(), is_alive(), terminate(), exitcode. |
Pool(processes=None) |
A group of workers. map(), imap(), imap_unordered(), starmap(), apply_async(). Use it as a context manager. |
Queue, JoinableQueue, SimpleQueue |
Process-safe queues for passing objects between processes. |
Pipe() |
Two connected endpoints for sending objects. |
Value, Array |
Small pieces of shared memory (a C number or array), protected by a lock. |
shared_memory.SharedMemory |
A named block of memory for large buffers (a submodule). |
Manager() |
A server process that hosts shared lists, dicts, locks and more, used through proxies. |
Lock, Event, Semaphore, Barrier |
Process-level versions of the threading primitives. |
cpu_count(), current_process(), active_children() |
Information about the machine and processes. |
get_context(method), set_start_method() |
Choose how new processes start: "fork", "spawn" or "forkserver". |
multiprocessing.dummy |
The same API backed by threads. |
Pool.map(func, iterable, chunksize=None) returns a list of results in the same order as the inputs. An exception raised in a worker is re-raised in the caller.
Usage notes
- Guard the entry point. Put process-starting code under
if __name__ == "__main__":. With thespawnstart method (the default on Windows and macOS) and withforkserver(the default on Linux from Python 3.14), each child imports your main module again; without the guard it would start processes recursively. - Everything crossing the boundary is pickled. Functions must be defined at module level (no lambdas or nested functions); arguments must be picklable (no open files, sockets or locks).
- Memory is not shared. A child that appends to a list changes only its own copy. Return results, or use a queue, a
Value, shared memory or aManager. - Overhead is real. Starting processes and pickling data costs milliseconds. Parallelism pays off for work that takes much longer than the transfer; for tiny tasks a plain loop wins. Use
chunksizeor chunk the work yourself. - Always close pools.
with Pool() as pool:terminates workers on exit; callclose()andjoin()if you wait forapply_async()jobs. - Daemon processes cannot create children, and are killed when the parent exits.
- Use
concurrent.futures.ProcessPoolExecutorfor a friendlier API with futures; it is built on this module. - Do not use it for I/O waiting. Threads or
asyncioare lighter for that. - Interactive sessions and notebooks often cannot pickle functions defined there under
spawn; put the worker functions in a module file.
Related
threading,concurrent.futures,asyncio,queue,pickleos- Lessons: Concurrency, Performance