Loky

Loky is a drop-in replacement for Python’s concurrent.futures.ProcessPoolExecutor that focuses on robustness across platforms and Python versions. It provides a deadlock-free process pool that recovers gracefully from worker crashes and returns meaningful errors instead of hanging.

Originally developed as the parallel backend for joblib (and used by scikit-learn), loky adds a reusable, dynamically resizable executor singleton, consistent fork/exec spawn behavior, and transparent cloudpickle integration so you can dispatch lambdas and interactively defined functions without the usual if __name__ == "__main__" boilerplate.

What You Get

• A deadlock-free ProcessPoolExecutor replacement that survives worker crashes
• A reusable, dynamically resizable executor singleton with idle-timeout shutdown
• Consistent fork/exec spawn behavior for safer third-party library interactions
• Transparent cloudpickle integration for lambdas and __main__-defined functions
• No need for if __name__ == "__main__" guards, including on Windows

Common Use Cases

• Serving as the process-based parallel backend for joblib and scikit-learn
• Running embarrassingly parallel CPU-bound work across worker processes
• Reusing a warm worker pool across many successive parallel calls
• Dispatching lambdas or interactively defined functions to subprocesses

Under The Hood

Architecture - Loky reimplements ProcessPoolExecutor in process_executor.py, layering on a reusable_executor.py that manages a resizable singleton pool with idle-timeout shutdown. A backend package abstracts OS process spawning and inter-process queues, while cloudpickle_wrapper.py swaps the default pickler for cloudpickle so closures and __main__ functions serialize correctly. The executor monitors worker liveness and converts crashes into BrokenProcessPool-style exceptions rather than deadlocking.

Tech Stack - Pure Python built on concurrent.futures and multiprocessing, with optional cloudpickle for extended serialization. It supports POSIX and Windows across a wide range of Python versions and is validated on Azure Pipelines CI.

Code Quality - The code is organized around clear modules (_base, process_executor, reusable_executor, backend, initializers) and carries an extensive, long-lived test suite that stresses crash recovery and resizing. As joblib’s production backend it is battle-tested across the scientific Python ecosystem.

API Design - The public API mirrors concurrent.futures (get_reusable_executor, submit, map), so existing users adopt it with minimal changes. The reusable-executor helper and automatic cloudpickle handling remove common friction points, making parallelism ergonomic without extra boilerplate.