dill

dill is a drop-in replacement for Python’s built-in pickle module that can serialize a much wider range of Python objects, including lambdas, nested functions, closures, generators’ state, and even entire interpreter sessions. Where pickle fails on functions defined in a REPL or on complex class hierarchies, dill succeeds by tracing and reconstructing the full object graph.

Because it mirrors pickle’s API (dumps/loads), existing code can adopt it with a single import change. It is widely used as the underlying serialization layer for distributed and parallel computing frameworks that need to ship arbitrary callables across process or network boundaries.

What You Get

• Drop-in dumps/loads API that mirrors the standard pickle module
• Serialization support for lambdas, closures, nested functions, and generators’ state
• Session save/restore (dill.session) to checkpoint and resume an entire interpreter state
• Source code introspection (dill.source) to extract source from live functions and classes
• Pickle-tracing diagnostics (dill.detect) to debug what is being serialized and why

Common Use Cases

• Shipping arbitrary Python callables (including lambdas) across processes in multiprocessing or pathos-based parallel computing
• Checkpointing long-running interactive or notebook sessions so work can resume later
• Serializing closures and dynamically generated classes for distributed task queues
• Diagnosing pickling failures in complex object graphs via tracing utilities

Under The Hood

Architecture dill is organized around _dill.py, which registers custom copyreg/pickler dispatch functions for each unsupported type (functions, cells, code objects, modules) so that the standard pickle.Pickler/Unpickler machinery can be extended without being replaced. session.py builds on this to snapshot an entire module’s or interpreter’s global namespace, while source.py and detect.py provide independent introspection layers that walk the same object graphs for source extraction and trace diagnostics respectively. This lets the core serialization logic, session handling, and debugging tooling evolve mostly independently. Tech Stack The library is pure Python (100% per GitHub’s language breakdown) with zero required runtime dependencies beyond setuptools, and optional extras (objgraph, gprof2dot, pyreadline) gated behind pip extras for diagnostics. It supports Python 3.10+ and ships both setup.py and pyproject.toml for PEP 517 builds. Code Quality The dill/tests/ directory contains dozens of targeted test modules (test_classdef.py, test_functions.py, test_nested.py, test_moduledict.py, etc.), each exercising one category of previously-unpicklable object, runnable via python -m dill.tests. Naming is consistent with the standard library’s pickle conventions, and the codebase has a long, incrementally-maintained history (28 releases, 1000+ commits) rather than large rewrites. API Design The public API is deliberately a superset of pickle’s (dumps/loads/dump/load), so existing pickle-based code adopts dill via a single import change, and settings like recurse/byref/fmode are exposed as ordinary keyword arguments rather than requiring new concepts.