Python Multiprocessing: ProcessPoolExecutor, Start Methods, Results, and Failures

Process Parallelism

Multiprocessing runs work in separate interpreter processes. It is useful for independent CPU-bound Python calculations that can be serialized and are large enough to justify process startup and data-transfer costs.

After this lesson, you can protect process creation with the main guard, map a top-level function through ProcessPoolExecutor, collect exceptions from futures, and explain why start-method assumptions and shared mutable state make code fragile across platforms.

Concurrency Choice

Measure the workload. Process parallelism can be slower when arguments and results are large or each task is too small.

Workload	Starting Point
Many blocking I/O calls	ThreadPoolExecutor or asyncio
Independent CPU-heavy Python functions	ProcessPoolExecutor
Tiny fast operations	Ordinary sequential loop

Main Guard

Child processes must be able to import the module without creating another pool recursively. Put process-starting code behind if __name__ == "__main__" and keep submitted functions importable at module level.

Square Values in Processes

from concurrent.futures import ProcessPoolExecutor

def square(value: int) -> int:
    return value * value

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=2) as pool:
        print(list(pool.map(square, [2, 3, 4])))

Output

[4, 9, 16]

The context manager shuts down worker processes after all mapped results are collected.

Serializable Boundary

Arguments, return values, and submitted callables cross a process boundary through serialization. Open connections, locks from another context, local functions, and many extension objects cannot be transferred safely. Pass compact data and create process-local resources inside the worker when needed.

Future Failures

A submitted task failure is raised when result() is read. Keep a stable input identifier with each future so logs can name the failed unit without printing private payloads.

Associate Results with Inputs

from concurrent.futures import ProcessPoolExecutor, as_completed

def reciprocal(value: int) -> float:
    return 1 / value

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        pending = {pool.submit(reciprocal, value): value for value in [2, 0]}
        for future in as_completed(pending):
            value = pending[future]
            try:
                print(value, future.result())
            except ZeroDivisionError:
                print(value, "failed")

Completion order can vary, so this example intentionally does not claim one exact output order.

Start Methods

Windows uses spawn. Python 3.14 changed the POSIX default away from fork; fork is no longer the default on any platform. Libraries should not force one global method without documenting the requirement and allowing the application to provide a context.

Process Boundaries

Avoid multiple workers writing the same file or row without coordination.
Set bounded worker counts from CPU, memory, and downstream capacity.
Use timeouts and cancellation as operational controls, not as proof that arbitrary code stopped safely.
Test process code as a real script, not only inside an interactive shell.

Select a Concurrency Model

0 of 2 checked

Q1. Which task most clearly fits a process pool?

Independent CPU-heavy calculations One tiny addition Waiting for a single keyboard input

Q2. Why use the main guard?

To prevent recursive process creation during module import To enable f-strings To sort results

Try this next

Measure a Process Pool

0 of 2 completed

Compare sequential and parallel work
Time a sufficiently large CPU calculation with one loop and a bounded process pool, including startup and result collection.
Design the transfer shape
Replace a non-serializable database connection argument with a compact record identifier and process-local setup.

Previous Next