pickleshare

PickleShare is a minimal, file-based key-value store for Python that behaves like the standard library’s shelve module, but with an important twist: it supports concurrent access from multiple processes safely. Instead of a single monolithic file, PickleShare stores each key as an individual pickled file inside a directory, so simultaneous readers and writers never corrupt a shared file, and changes made by one process become visible to others immediately.

The library exposes a single class, PickleShareDB, that implements Python’s MutableMapping interface so it can be used exactly like a dictionary — db['key'] = value, iteration, len(), and deletion all work as expected. It also adds a handful of convenience methods for IPC-style workflows: hset/hget/hdict for hashed sub-collections, waitget for polling until another process writes a key, and getlink for attribute-style nested access. Originally built to back IPython’s caching and object-persistence needs, it remains useful today for lightweight local storage and simple inter-process communication.

What You Get

• Drop-in dict-like API via MutableMapping (db[key], len(db), iteration, deletion)
• Multi-process-safe storage — each key is its own file, so writers don’t clobber each other
• Hashed sub-collections (hset/hget/hdict/hcompress) for grouping many small values efficiently
• waitget polling helper for simple cross-process synchronization
• PickleShareLink for attribute-style access to nested keys
• Small single-file implementation with no required third-party dependencies on modern Python

Common Use Cases

• Sharing small pieces of state or cached results between multiple Python processes on the same machine
• Lightweight local caching where a full database or shelve file isn’t worth the complexity
• IPC-style coordination — one process polls waitget while another writes a result key
• Persisting IPython/Jupyter session state (the library’s original use case)

Under The Hood

Architecture PickleShare is implemented as a single module (pickleshare.py) built around one class, PickleShareDB, which subclasses collections.abc.MutableMapping and wraps a pathlib.Path root directory. Each dictionary key maps directly to a file path under that root: __getitem__/__setitem__ read and write individual pickle files rather than one shared database file, which is what gives the library its multi-process safety — there is no single file two processes can race to corrupt. A small in-memory cache (keyed by file modification time) avoids re-reading unchanged files, and hset/hget/hdict/hcompress layer a hashed-bucket scheme on top of the same file-per-key model to keep large collections of small values from producing one file per item. PickleShareLink is a thin companion class that maps attribute access onto nested keys via __getattr__/__setattr__. The design is intentionally shallow — there is no query layer, no schema, and no transaction support — so the abstraction is easy to reason about but would need real work to extend (e.g. adding atomic multi-key writes).

Tech Stack The library has no required runtime dependencies on modern Python — it uses only the standard library (os, stat, time, pickle, errno, pathlib, collections.abc). A pathlib2 backport is pulled in via extras_require only for legacy Python 2.6/2.7/3.2/3.3 environments. Packaging is a plain setuptools setup.py that extracts __version__ by parsing the source file rather than importing it. Tests run under pytest (using the tmpdir fixture) and are wired to Travis CI via .travis.yml, though the project has seen no commits since late 2023.

Code Quality The test suite (test_pickleshare.py) covers the core dict-like operations, hashed get/set, and a stress test that hammers the store from simulated concurrent writers, but it is small and has no coverage reporting or type checking. The code still carries visible Python 2/3 compatibility shims (try/except ImportError for pathlib/cPickle, a string_types tuple) and at least one bare except: clause in __getitem__ that swallows all read errors as a KeyError, which is convenient but can mask real bugs. There are no type hints, no linter configuration, and naming is terse and dated (hroot, hfile, fil) rather than following modern PEP 8 conventions. Overall it reads as functional, lightly-tested legacy code rather than a actively-maintained, rigorously-typed project.

What Makes It Unique PickleShare’s one real technical idea is storing each key as its own file instead of one shared database file, which sidesteps the classic shelve/dbm problem of concurrent writers corrupting a single backing file — filesystem-level atomicity of individual file writes stands in for a real locking or transaction layer. Beyond that trade-off, the library is deliberately unambitious: it favors a tiny, dependency-free implementation over the features of a real embedded database, which is exactly why it has remained a stable, low-maintenance dependency inside IPython and similar tools for over a decade rather than a novel piece of engineering in its own right.