billiard

billiard is a standalone fork of Python’s stdlib multiprocessing package, carrying forward fixes and improvements that were pulled from python-trunk plus additional patches contributed by the Celery maintainers. It reimplements the full multiprocessing API surface (Process, Pool, Queue, managers, synchronization primitives) while adding execv-avoidance on POSIX, extra worker-pool controls, and more resilient error reporting across process boundaries.

The project exists specifically to back Celery’s worker pool: its Pool implementation adds soft/hard per-task time limits, maxtasksperchild-based worker recycling, and detailed exception propagation (WorkerLostError, MaybeEncodingError) that the stdlib pool doesn’t provide. It ships a small C extension for platform-specific semaphore and Windows API support, falling back gracefully to the built-in _multiprocessing module when a C compiler isn’t available.

What You Get

• A drop-in replacement for Python’s multiprocessing.Pool with soft and hard per-task time limits (SIGUSR1-based soft limit, SIGKILL-based hard limit)
• maxtasksperchild worker recycling with tracked restart timing and on_process_exit callbacks
• Cross-process exception propagation via einfo.ExceptionInfo, plus dedicated WorkerLostError and MaybeEncodingError types for diagnosing worker crashes
• Context-based selection of fork, spawn, and forkserver process-starting strategies, each implemented as a dedicated popen_* module
• Optional C extension (_billiard) for semaphore and Windows API support, with automatic fallback to pure-Python/stdlib _multiprocessing if compilation fails
• The same Manager/proxy-object API as stdlib multiprocessing for sharing state (lists, dicts, namespaces) across worker processes

Common Use Cases

• Running Celery worker pools that need per-task time limits and safe worker recycling under long-running or memory-leaking tasks
• Executing CPU-bound Python work in parallel across processes when the GIL rules out threads
• Isolating unreliable or crash-prone code (e.g. calling into flaky C extensions) in a subprocess so a crash doesn’t take down the whole application
• Building custom task-execution frameworks that need finer-grained control over process lifecycle than stdlib multiprocessing exposes

Under The Hood

Architecture billiard mirrors stdlib multiprocessing’s layered design: an abstract context layer (context.py) selects a platform-specific process-starting strategy — popen_fork.py, popen_spawn_posix.py, popen_spawn_win32.py, or popen_forkserver.py — each implementing the same Popen interface for a different bootstrapping mechanism (fork, spawn, forkserver). process.py’s BaseProcess sits on top of whichever Popen is selected, and pool.py builds billiard’s main value-add on top of that: a worker-pool orchestrator with dedicated TaskHandler and ResultHandler threads, retry/timeout bookkeeping, and maxtasksperchild-driven worker replacement. managers.py layers proxy objects (shared lists, dicts, namespaces) over the same process abstraction, and reduction.py/heap.py handle the pickling and shared-memory plumbing IPC depends on. Because pool.py, managers.py, and forkserver.py all depend on the same bootstrapping semantics established in init.py and context.py, a change to the core Process/Popen contract ripples through every layer above it.

Tech Stack Pure Python 3.7+ with an optional C extension (Modules/_billiard/*.c) compiled via setuptools.Extension for POSIX semaphore support and native Windows API bindings, falling back to the pure-Python/stdlib _multiprocessing module when no compiler is available. The package has zero runtime third-party dependencies — only dev/test tooling (tox, pytest, pytest-cov, flake8, psutil) is declared. CI runs the unit-test matrix across Python 3.10 through 3.14 via tox-gh-actions and tox-docker, plus CodeQL security scanning and codespell for typo checks.

Code Quality Tests live under t/unit (covering pool, context, spawn, dummy, values, einfo, and Windows-specific behavior) and t/integration, which adapts CPython’s own multiprocessing test suite. Everything runs through pytest/tox with flake8 linting and a pre-commit config for style enforcement. Error handling is explicit and deliberate — pool.py defines and raises dedicated exception types (WorkerLostError, MaybeEncodingError, SoftTimeLimitExceeded, TimeLimitExceeded) rather than letting worker crashes fail silently. There are no type annotations or static type checking configured, consistent with its origins as a fork of pre-typing-era stdlib code.

What Makes It Unique billiard’s differentiators are all production-hardening features layered on top of stdlib multiprocessing rather than new algorithms: per-task soft and hard time limits enforced via signals, maxtasksperchild worker recycling with restart-timing tracking and exit callbacks, and an exaggerated ExceptionInfo/traceback serialization scheme (einfo.py) built specifically so a worker’s traceback survives being pickled back to the parent process — a capability Celery leans on heavily for surfacing task failures. It is a maintenance fork built to stay ahead of (or diverge deliberately from) upstream CPython multiprocessing on exactly the reliability edges that a production task-queue worker pool needs.