serde-pickle

serde-pickle is a Rust crate for parsing and generating Python pickle streams, built on top of Serde’s generic serialization framework. It decodes all pickle protocols (0 through 5) and encodes protocol 2 or protocol 3, so data pickled by CPython can be read directly by a Rust program and vice versa without a Python subprocess in between.

Because pickle can represent Python types that Serde’s generic data model can’t express directly - arbitrary-precision integers, sets, and frozensets used as dict keys - the crate also exposes a dedicated Value/HashableValue enum with value_to_vec/value_from_slice functions for full-fidelity round-tripping, alongside the generic to_vec/from_slice functions that work with any type deriving Serialize/Deserialize.

What You Get

• Generic serde to_vec/from_slice functions for reading and writing pickle streams from any type that derives Serialize/Deserialize
• A dedicated Value/HashableValue enum plus value_to_vec/value_from_slice for full-fidelity access to Python’s pickle types, including big integers, sets, and frozensets
• Support for decoding all pickle protocols (0 through 5) and encoding protocol 2 or protocol 3
• SerOptions/DeOptions builders for toggling protocol quirks: legacy enum representation, restore-state retention, and unresolved-global/recursive-structure handling

Common Use Cases

• Reading Python-pickled datasets or model artifacts from a Rust service without round-tripping through a Python subprocess
• Building Rust tooling that interoperates with existing Python pickle-based caches, checkpoints, or IPC payloads
• Converting pickle streams to and from JSON for debugging or migration, as demonstrated by the crate’s example binary
• Serializing Rust structs into pickle format for consumption by downstream Python code

Under The Hood

Architecture The crate splits cleanly into ser.rs (a Serializer struct writing pickle opcodes), de.rs (a two-stage deserializer: an internal intermediate Value enum with MemoRef/Global variants that resolves memo-table references and module-global reductions before materializing into the public value::Value type), value.rs (the public Value/HashableValue enums modeling every Python builtin pickle type), value_impls.rs (serde Serialize/Deserialize impls bridging Value to and from serde’s generic data model), consts.rs (pickle opcode byte constants), and error.rs (Error/ErrorCode enums distinguishing Io/Eval/Syntax failures with byte-offset context). de::Deserializer reads opcodes sequentially off a BufReader-wrapped IterRead adapter, pushes intermediate values onto an internal stack mirroring the pickle VM, resolves memo indirection via a hash-backed memo table, then either implements serde’s Deserializer trait to feed a caller’s derived struct or converts fully into value::Value. Because both paths share the same intermediate stack machine, changes to it require coordinated updates across de.rs and value_impls.rs.

Tech Stack Rust 2018 edition, MSRV 1.63. Core dependencies: serde 1.0.209 (the trait framework this crate implements Serializer/Deserializer against), byteorder 1.5.0 (big/little-endian integer encoding for pickle’s binary opcodes), num-bigint/num-traits (arbitrary-precision integers for Python’s unbounded int), and iter-read (adapts an iterator into std::io::Read for streaming decode). An optional criterion dependency is gated behind a criterion-bench feature for benchmarking. Dev-dependencies include serde_derive, serde_json (used by the example binary’s JSON conversion commands and by tests), and quickcheck for property-based testing. No async runtime and no unsafe-heavy FFI - a pure safe-Rust binary protocol implementation. CI runs cargo fmt --check, cargo clippy, and cargo test against both the pinned MSRV and stable Rust.

Code Quality The test suite (test/mod.rs, included via a #[path] attribute under #[cfg(test)]) uses serde_derive on sample structs and enums to verify round-trip correctness of to_vec/from_slice, plus test/arby.rs for Arbitrary/quickcheck-driven property testing of Value. test/data/ ships real Python-pickled fixture files spanning protocols 0-5 and Python 2/3, including recursive and unresolvable-global edge cases - genuine interoperability testing, not just self-consistency. Error handling is fully typed: no panics or unwraps on malformed input in the decode path, with failures surfacing as Error::Eval(ErrorCode, byte_offset) or Error::Syntax(ErrorCode) carrying actionable context. A fuzz/ directory with a cargo-fuzz harness targets the decoder directly against malformed byte input, and CI enforces formatting and linting on every push.

API Design The crate’s core ergonomic decision is offering both a generic serde-compatible API (to_vec/from_slice, usable with any #[derive(Serialize, Deserialize)] type with no extra glue) and a dedicated Value enum plus value_to_vec/value_from_slice pair for callers who need full pickle type fidelity that Serde’s generic model can’t express. SerOptions/DeOptions builder methods surface protocol quirks - legacy enum representation, restore-state retention, unresolved-global and recursive-structure fallback - as named opt-in toggles rather than positional booleans, and the crate’s docs explicitly call out which pickle features (recursive PERSID/EXT opcodes, out-of-band protocol 5 buffers) are intentionally unsupported rather than silently mishandled.