cloudpickle

cloudpickle extends Python’s standard pickle module to serialize objects that pickle cannot handle on its own — lambda expressions, functions and classes defined interactively in __main__ (a script, shell, or Jupyter notebook), and closures capturing local state. It was built for cluster computing, where Python code and data need to travel over the network to remote workers, and today underpins distributed frameworks like Dask, PySpark, and Ray for exactly that reason.

What You Get

• cloudpickle.dumps/loads — a pickle-compatible API for serializing objects standard pickle cannot, including lambdas and closures
• Support for pickling functions and classes defined interactively in __main__ (scripts, REPL sessions, Jupyter notebooks)
• A register_pickle_by_value/unregister_pickle_by_value API to force serialization-by-value for specific modules under active development
• Correct handling of nested functions and closures that capture surrounding local variables
• Compatibility layer so output can be loaded with the standard library’s pickle.loads

Common Use Cases

• Distributed computing frameworks (Dask, PySpark, Ray) shipping user-defined functions and closures to remote worker processes
• Serializing interactively defined functions/classes from a Jupyter notebook or REPL session so they can run on a separate worker
• Checkpointing or transmitting callables in task-queue systems where the callable was defined in __main__ rather than an importable module
• ML training pipelines that pass custom loss functions, lambdas, or locally-defined model classes to worker processes

Under The Hood

Architecture — The library is a thin, focused patch over the CPython pickle protocol: cloudpickle.py implements custom Pickler subclass reducers that intercept function, class, and closure-cell objects before the standard reducer would fail on them, detecting whether a given object was defined in an importable module (serialize by reference, like stock pickle) versus interactively or dynamically (serialize by value — capture bytecode, globals, and closure cells). cloudpickle_fast.py and __init__.py are thin wiring/compat shims. Tech Stack — Pure Python (99.9% of the codebase) with zero runtime dependencies, targeting compatibility across CPython and PyPy; the test suite is run via tox across supported Python versions. Code Quality — A large tests/ directory (cloudpickle_test.py, test_backward_compat.py, cloudpickle_file_test.py, cloudpickle_ipykernel_test.py) covers core serialization, cross-version backward compatibility, and notebook-specific edge cases; the README explicitly documents known limitations (e.g. functions pickled by value containing local import statements can fail), which is a good maintainability signal. API Design — The API is deliberately a drop-in replacement for pickle.dumps/pickle.loads with no new concepts to learn for basic use; advanced behavior (register_pickle_by_value) is opt-in and only needed for the specific case of live-reloading modules on distributed workers.