multiprocess

multiprocess is a fork of the Python standard library’s multiprocessing module, maintained as part of the pathos framework for heterogeneous computing. It reimplements the same Process, Pool, Queue, Manager, and synchronization primitives as stdlib multiprocessing, but swaps the default pickle-based serializer for dill, which can serialize lambdas, nested functions, class instances, and other objects that the standard pickle protocol rejects.

Because the public API mirrors multiprocessing almost exactly, existing code can typically switch over with a single import change (import multiprocess as multiprocessing). The project maintains parallel implementations per Python version (py3.10 through py3.15, plus PyPy 3.10/3.11) so it can track CPython’s own multiprocessing internals as they evolve, including a small C extension for platform-specific semaphore/shared-memory support.

What You Get

• Full drop-in replacement for multiprocessing’s Process, Pool, Queue, Pipe, and Manager APIs
• dill-based serialization that can pickle lambdas, closures, nested functions, and many class instances
• Synchronization primitives (Lock, RLock, Semaphore, Condition, Event, Barrier) matching threading’s API
• SyncManager-based shared objects (list, dict, Namespace) served from a background manager process
• Per-Python-version source trees (py3.10-py3.15, PyPy) tracking CPython’s own multiprocessing changes
• dummy submodule providing a thread-backed API-compatible fallback, same as multiprocessing.dummy

Common Use Cases

• Parallelizing CPU-bound Python workloads across processes with Pool.map / map_async
• Passing closures or lambdas as worker functions to a process pool, which plain multiprocessing cannot pickle
• Sharing mutable state (lists, dicts, custom objects) between worker processes via a SyncManager
• Migrating existing multiprocessing code to gain broader object serialization with a one-line import swap
• Building on top of pathos, which uses multiprocess as its underlying process-pool engine

Under The Hood

Architecture multiprocess mirrors the layered structure of CPython’s own multiprocessing module: process.py defines the Process abstraction, context.py provides BaseContext/DefaultContext objects that select fork/spawn/forkserver start methods and expose the public factory methods (Pool, Queue, Manager, Lock, etc.), reduction.py owns the serialization layer that swaps in dill wherever the stdlib uses pickle, pool.py implements the worker-pool scheduling and result-dispatch logic, and managers.py implements the SyncManager/BaseProxy machinery that serves shared list/dict/Namespace objects from a background manager process. Because the fork tracks CPython release by release, this same layered design is duplicated across separate py3.10-py3.15 (and PyPy) source trees rather than being version-abstracted in a single codebase, which keeps each version’s behavior faithful to its corresponding stdlib release at the cost of code duplication across trees.

Tech Stack The project targets Python and PyPy 3.10+, depends on dill>=0.4.1 for its enhanced pickling, and builds via a PEP 517 setuptools>=42 backend declared in pyproject.toml with the actual build logic in setup.py. A small C extension module (_multiprocess) is compiled per Python-version directory to back platform-specific primitives such as semaphores, mirroring the _multiprocessing C extension in CPython. CI runs on Travis across Python 3.10 through 3.15 (including a free-threaded 3.15t build) and PyPy 3.10/3.11, with a coverage matrix toggled per job via COVERAGE/DILL/FORK_PY environment flags.

Code Quality The tests package (multiprocess.tests) is a substantial port of CPython’s own multiprocessing test suite, split into fork/forkserver/spawn variants plus dedicated preload and main-handling test modules, and is runnable directly via python -m multiprocess.tests. A .coveragerc scopes coverage collection to the multiprocess and multiprocess.dummy packages while excluding the C extension, examples, and tests themselves, and Travis exercises this suite across every supported interpreter and start method. No type annotations, mypy configuration, or linter/formatter config (ruff, flake8, black) were found in the repository, so static type and style enforcement is limited to what the CI test matrix catches at runtime.

API Design The library’s central design goal is API parity with the standard library: the same Process, Pool, Queue, Manager, and synchronization primitive names and signatures are reused, so most adopters migrate with a single import swap (import multiprocess as multiprocessing) rather than learning a new interface. This keeps boilerplate at zero beyond the import change, and the dummy submodule extends that same familiar API to a thread-backed implementation for cases where processes aren’t needed. The tradeoff is that the library introduces no new abstractions of its own beyond the dill-powered serialization it adds transparently under the hood.