52 · Multiprocessing: real parallelism for CPU-bound work
Each Process is a full Python interpreter — true CPU parallelism that sidesteps the GIL entirely. Learn Process, Pool.map, fork vs spawn, and the mandatory __main__ guard.
`multiprocessing` spawns real OS processes — each with its own Python interpreter and memory space. This sidesteps the GIL completely, enabling true CPU parallelism across all cores. The trade-off: everything passed between processes must be serialized (Python's `marshal` protocol handles this), adding overhead that makes small tasks slower in parallel than sequential. `Pool(n).map(fn, items)` is the one-liner that turns embarrassingly parallel work into genuine speedup.
Without this:
Without `multiprocessing` (or C/numpy vectorisation) you cannot use more than one CPU core for Python computation — the GIL ensures that. A 4-core machine running a pure-Python number crunch uses 25% of its hardware. `Pool(4)` can bring that to near 100%.
Why a new process instead of a new thread?
Each multiprocessing.Process is a full OS process: it gets its own Python interpreter, its own GIL, its own heap, and its own copy of every module import. This isolation is what makes true parallelism possible — two processes run Python bytecode simultaneously on two separate CPU cores with no GIL contention.
The costs:
- Startup time — forking or spawning a process takes ~30–100 ms. Don't parallelize tasks shorter than this.
- Memory — each process copies the parent's memory (on Unix via copy-on-write, but writes diverge). 4 workers × 1 GB model = 4 GB RAM.
- Serialization — all arguments and return values cross process boundaries as serialized bytes. Functions and large numpy arrays must all be serializable.
Fork vs spawn
Unix gives you two ways to create a child process:
- fork (default on Linux/macOS < 3.12) — the OS copies the parent process's entire memory map instantly using copy-on-write. Fast, but inherited open file handles, database connections, and thread locks can cause silent corruption in the child.
- spawn (default on Windows, default everywhere from 3.12+) — the child starts with a clean Python interpreter and re-imports everything from scratch. Slower but safe. Preferred for production.
multiprocessing.set_start_method("spawn") forces spawn on any platform.
The if __name__ == "__main__": guard
With the spawn start method, Python spawns a new interpreter and imports your script to create the child process. If your script immediately calls Pool(4).map(...) at module level, the child process does the same — spawning 4 more children, which each spawn 4 more, forever. The guard stops this:
if __name__ == "__main__": # only runs in the parent, not in spawned children
with Pool(4) as pool:
results = pool.map(my_fn, range(100))
On Linux with fork, the guard is technically optional because forked children don't re-execute module-level code — but writing it anyway is a best practice (your code stays portable to Windows and Python 3.12+).
Notice `p1.start(); p2.start()` before either `.join()` — this lets both processes run concurrently. If you called `p1.start(); p1.join()` before even creating `p2`, the processes would run sequentially.
`Pool(4)` used as a context manager automatically calls `.terminate()` and `.join()` on exit. `pool.map(fn, iterable)` is a synchronous parallel map — it blocks until all results are ready. For large iterables, `pool.imap(fn, iterable)` streams results lazily. `pool.starmap(fn, [(a1, b1), ...])` passes multiple arguments per task.
The tl;dr on start methods: use `spawn` in any code you want to run on Windows or Python 3.12+. If you're on Linux and spawn is too slow, `forkserver` is a safer alternative to `fork` for multi-threaded parents.
The expected speedup is `n_cores - epsilon` — not exactly `n_cores` because process startup, IPC serialization overhead, and OS scheduling all eat into the theoretical maximum. For tiny tasks (< ~10 ms each), overhead dominates and you may see a slowdown rather than a speedup.
What happens if you forget the `if __name__ == '__main__':` guard when using `multiprocessing.Pool` with the `spawn` start method on Windows?
- `multiprocessing.Process` creates a real OS process with its own interpreter and GIL — enabling true CPU parallelism. Each process starts with either `fork` (Linux, fast but unsafe for multi-threaded parents) or `spawn` (Windows/3.12+, slow but safe).
- Always wrap the entry point in `if __name__ == '__main__':` — without it, `spawn`-based child processes re-import the module and re-execute the `Pool(...)` call, creating an infinite spawn loop.
- `Pool(n).map(fn, iterable)` is the high-level API for embarrassingly parallel work — distributes tasks across n workers and collects results in order.
- Everything crossing process boundaries must be serializable. IPC overhead makes parallelism counterproductive for tasks shorter than ~10 ms.
- `multiprocessing` does NOT work in the Pyodide browser sandbox (no fork/spawn in WebAssembly). Run multiprocessing code locally.
scikit-learn's `n_jobs=-1` parameter (in `GridSearchCV`, `RandomForestClassifier`, `cross_val_score`, etc.) calls `joblib.Parallel(n_jobs=-1)`, which uses multiprocessing under the hood on most platforms. XGBoost and LightGBM's `n_jobs` parameter, parallel feature extraction pipelines, and batch offline inference over large datasets all rely on `multiprocessing.Pool`. When you run `RandomForestClassifier(n_estimators=200, n_jobs=4)`, each of the 4 workers builds 50 trees independently — a textbook embarrassingly parallel workload.
If you remove it: Without multiprocessing, every CPU-bound ML workload is single-threaded. Grid search over 1000 hyperparameter combinations takes 10 hours instead of 2.5 on a 4-core laptop. Cross-validation over 10 folds runs sequentially instead of in parallel. All the `n_jobs` parameters in scikit-learn become meaningless.