ormsgpack

ormsgpack is a MessagePack serialization library for Python implemented as a Rust extension via PyO3, forked from the widely-used orjson JSON library. It exposes just two functions, packb and unpackb, but handles a far wider range of native Python types than the stdlib msgpack package out of the box: dataclasses (including slotted and frozen variants), datetime/date/time objects, UUIDs, enums, numpy scalars and arrays, and Pydantic models, all without requiring custom encoder callbacks for the common cases.

Behavior is controlled through a small set of bitwise-composable OPT_* integer constants rather than a large parameter surface, covering things like sorting keys deterministically, encoding aware datetimes as MessagePack timestamp extensions, allowing non-string dict keys, and selectively “passing through” specific types (big ints, tuples, subclasses, enums) to a user-supplied default callback instead of ormsgpack’s built-in handling. It also provides Ext and Fragment wrapper types for working directly with MessagePack extension types and pre-serialized byte fragments, letting callers splice already-encoded MessagePack data into a new payload without a decode/re-encode round trip.

The project targets CPython, PyPy, and GraalPy, ships prebuilt wheels for the major platforms, and is benchmarked continuously via CodSpeed to catch performance regressions across releases. It follows semantic versioning and is dual-licensed under Apache-2.0 and MIT.

What You Get

• A packb/unpackb API that serializes dataclasses, datetimes, dates, times, UUIDs, enums, numpy scalars/arrays, and Pydantic models natively, without writing custom encoders for the common cases
• Bitwise-composable OPT_* options (OPT_SORT_KEYS, OPT_NON_STR_KEYS, OPT_DATETIME_AS_TIMESTAMP_EXT, OPT_UTC_Z, OPT_NAIVE_UTC, OPT_OMIT_MICROSECONDS, and OPT_PASSTHROUGH_* variants) to control encoding behavior precisely
• Ext and Fragment types for round-tripping MessagePack extension types and splicing pre-serialized byte fragments into a payload without decoding and re-encoding them
• MsgpackEncodeError and MsgpackDecodeError exception types (subclassing TypeError and ValueError respectively) for predictable error handling
• Prebuilt wheels for CPython, PyPy, and GraalPy across Linux, macOS, and Windows, plus full type stubs (py.typed) for static type checking
• Continuous performance benchmarking via CodSpeed on every PR, so serialization speed regressions are caught before release

Common Use Cases

• Replacing the stdlib msgpack package in services that need to serialize dataclasses, datetimes, or numpy arrays without hand-writing default encoders
• High-throughput API or RPC payload encoding where MessagePack’s compact binary format is preferred over JSON for bandwidth or parsing speed
• Caching serialized objects (e.g. Redis values) where Fragment lets previously-encoded MessagePack bytes be embedded directly into a new payload
• Serializing Pydantic models or numpy-heavy data pipelines directly to a binary wire format without a manual dict-conversion step
• Cross-language data interchange where a compact, well-specified binary format (MessagePack) is needed instead of pickling

Under The Hood

Architecture The crate is a cdylib PyO3 extension (src/lib.rs) that registers two module-level C functions, packb and unpackb, using the fast vectorcall calling convention (METH_FASTCALL | METH_KEYWORDS) and per-interpreter/no-GIL module slots on newer CPython versions. Serialization and deserialization are split into dedicated modules (src/serialize, src/deserialize) built on top of a msgpack module (src/msgpack) that defines per-type marker/encoding logic (array, map, bin, ext, bool, float, int, nil), an ffi module wrapping raw CPython/PyO3 internals for unicode and int handling, and a small state.rs holding per-module interned strings and cached type objects. Options are represented as a single bit-packed integer (src/opt.rs) validated against precomputed PACKB_OPT_MASK/UNPACKB_OPT_MASK constants at the C boundary, keeping the hot path free of per-call allocation for option parsing.

Tech Stack Written in Rust (edition 2021, MSRV 1.88) on top of pyo3 ^0.29 in extension-module mode, with ahash, chrono, half, itoa, serde, serde_bytes, simdutf8, and smallvec as focused, no-default-features dependencies chosen for encoding/decoding throughput rather than general convenience. The Python side is a thin wrapper (python/ormsgpack) with a full .pyi stub and py.typed marker; the build is orchestrated by maturin with python-source pointed at the python/ directory, and release profiles use fat-ish LTO (“thin”), single codegen unit, and stripped symbols for binary size and speed.

Code Quality Tests are Python-side pytest files under tests/, one file per data type (test_dataclass.py, test_datetime.py, test_numpy.py, test_pydantic.py, test_uuid.py, test_fragment.py, test_ext_type.py, and more), plus dedicated coverage for edge cases like circular references and concurrency. mypy runs in strict mode over the Python stub/wrapper, ruff lints the Python side, and rustfmt/clippy lint the Rust side, all wired into a dedicated Lint GitHub Actions workflow plus a CodSpeed workflow that runs the pytest-codspeed benchmark suite on every PR to catch performance regressions before merge.

What Makes It Unique Most Python MessagePack bindings mirror the stdlib msgpack package’s API and leave dataclasses, datetimes, numpy, and Pydantic models to caller-supplied default functions. ormsgpack, forked from orjson, instead treats those types as first-class citizens of the encoder itself, and exposes fine-grained OPT_* passthrough flags so callers can selectively opt specific types back out to a custom default when the built-in encoding isn’t what they want. The Fragment type is a similarly targeted feature: it lets already-encoded MessagePack bytes (e.g. from a cache) be spliced into a new payload without a decode/re-encode round trip, which general-purpose bindings don’t typically offer.