Skip to content
Tutorials Center
My learning

Topics

Python Reference ยท Standard Library: Concurrency, Network and Tools

The multiprocessing module

Reference for Python's multiprocessing module: Process, Pool.map, Queue, Value, Manager, start methods, pickling limits and the __main__ guard.

The multiprocessing module runs Python code in separate operating-system processes. Each process has its own interpreter and its own memory, so the global interpreter lock (GIL) is not a bottleneck: CPU-bound work can use every core. The price is that processes do not share ordinary objects. Arguments and results are copied between them by pickling, and shared state needs explicit tools such as queues, pipes, shared values or a manager.

For most jobs you do not create processes by hand. You build a Pool (or concurrent.futures.ProcessPoolExecutor) and map a function over your data; the pool splits the work, runs it in worker processes and returns the results in order.

Python
from multiprocessing import Pool

def square(n):
    return n * n

if __name__ == "__main__":
    with Pool() as pool:
        print(pool.map(square, range(5)))   # [0, 1, 4, 9, 16]

The in-browser runner cannot start processes (the module's C helper _multiprocessing is not available), so Process, Pool, Queue and Manager are shown read-only here. The examples that run below cover what is checkable in a single process: how data must be picklable, and how to structure code so it moves to a pool unchanged.

Try it

Python Functions to ship to workers must be picklable

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Split the work in chunks (single-process stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same chunks on a pool, as it would run in a normal Python environment (read-only):

Python
from multiprocessing import Pool

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.map(count_primes, chunks(range(2, 20000), 5000))
    print(sum(results))

What is in the module

Name Description
Process(target, args, kwargs, daemon) One child process. start(), join(), is_alive(), terminate(), exitcode.
Pool(processes=None) A group of workers. map(), imap(), imap_unordered(), starmap(), apply_async(). Use it as a context manager.
Queue, JoinableQueue, SimpleQueue Process-safe queues for passing objects between processes.
Pipe() Two connected endpoints for sending objects.
Value, Array Small pieces of shared memory (a C number or array), protected by a lock.
shared_memory.SharedMemory A named block of memory for large buffers (a submodule).
Manager() A server process that hosts shared lists, dicts, locks and more, used through proxies.
Lock, Event, Semaphore, Barrier Process-level versions of the threading primitives.
cpu_count(), current_process(), active_children() Information about the machine and processes.
get_context(method), set_start_method() Choose how new processes start: "fork", "spawn" or "forkserver".
multiprocessing.dummy The same API backed by threads.

Pool.map(func, iterable, chunksize=None) returns a list of results in the same order as the inputs. An exception raised in a worker is re-raised in the caller.

Usage notes

  • Guard the entry point. Put process-starting code under if __name__ == "__main__":. With the spawn start method (the default on Windows and macOS) and with forkserver (the default on Linux from Python 3.14), each child imports your main module again; without the guard it would start processes recursively.
  • Everything crossing the boundary is pickled. Functions must be defined at module level (no lambdas or nested functions); arguments must be picklable (no open files, sockets or locks).
  • Memory is not shared. A child that appends to a list changes only its own copy. Return results, or use a queue, a Value, shared memory or a Manager.
  • Overhead is real. Starting processes and pickling data costs milliseconds. Parallelism pays off for work that takes much longer than the transfer; for tiny tasks a plain loop wins. Use chunksize or chunk the work yourself.
  • Always close pools. with Pool() as pool: terminates workers on exit; call close() and join() if you wait for apply_async() jobs.
  • Daemon processes cannot create children, and are killed when the parent exits.
  • Use concurrent.futures.ProcessPoolExecutor for a friendlier API with futures; it is built on this module.
  • Do not use it for I/O waiting. Threads or asyncio are lighter for that.
  • Interactive sessions and notebooks often cannot pickle functions defined there under spawn; put the worker functions in a module file.