msgpack

msgpack (imported as github.com/vmihailenco/msgpack/v5) implements the MessagePack binary serialization format for Go, giving applications a drop-in, more compact and faster alternative to encoding/json for internal service-to-service payloads, caches, and storage. It encodes primitives, slices, maps, structs, time.Time, and interface{} values, and mirrors the ergonomics of the standard library with Marshal/Unmarshal and streaming Encoder/Decoder types.

Struct encoding is driven by msgpack struct tags — field renaming, aliases, omitempty, and a SetCustomStructTag escape hatch that lets the same struct definitions be reused with json or other tags. A CustomEncoder/CustomDecoder and Marshaler/Unmarshaler interface pair, plus a low-level Register extension API, cover types that need custom wire representations, and an msgpack extension-type mechanism preserves type information across the wire when needed.

The package leans on sync.Pool for both Encoder and Decoder instances and on build-tag-gated unsafe string/byte conversions to avoid copies on the hot path, alongside a reflect-based struct field cache keyed by type and tag so repeated encodes of the same struct shape stay cheap. It has been stable production infrastructure for the Go ecosystem for over a decade and is used as the wire format inside other open-source projects such as Uptrace’s Bun ORM tooling.

What You Get

• Marshal/Unmarshal + streaming API - one-shot Marshal/Unmarshal functions plus NewEncoder/NewDecoder for writing to and reading from io.Writer/io.Reader streams.
• Struct tag control - msgpack:"name", msgpack:"alias:other", and ,omitempty tags control field naming and omission without extra code.
• Custom encoding hooks - CustomEncoder/CustomDecoder and Marshaler/Unmarshaler interfaces, plus a Register API, for types needing bespoke wire formats.
• Extension type registry - RegisterExt support for encoding type information alongside values, so decoders can reconstruct the original type.
• Map key sorting and array-encoded structs - SetSortMapKeys for deterministic map output and UseArrayEncodedStructs to encode structs as arrays instead of maps for smaller payloads.
• Query API - a Decoder.Query method for extracting specific fields from a msgpack payload without fully decoding it into a struct.

Common Use Cases

• Internal RPC and service payloads - replacing JSON between Go microservices where payload size and encode/decode speed matter more than human readability.
• Cache and datastore values - storing structs in Redis, Memcached, or on-disk caches as compact MessagePack instead of JSON.
• Message queue envelopes - encoding job/event payloads for queues where consumers and producers are both Go services sharing struct definitions.
• Cross-language wire format - exchanging data with non-Go services (Node.js, Python, etc.) over a MessagePack-based protocol rather than a Go-specific binary format.
• Embedding inside other libraries - used as the underlying serialization layer inside other Go tools and ORMs (e.g. Uptrace’s Bun) that need a fast binary codec.

Under The Hood

Architecture The package is organized as a small set of top-level files rather than internal layers: encode.go/decode.go define the Encoder/Decoder types and the Marshal/Unmarshal entry points, types.go builds a reflect-based dispatch table (typeEncMap/typeDecMap, a structCache keyed by (reflect.Type, tag)) that resolves the right encoder/decoder function for a given Go type once and reuses it, and msgpcode/msgpcode.go isolates the raw MessagePack wire-format byte codes from the encoding logic above it. ext.go layers the extension-type registry on top of that dispatch table. Both Encoder and Decoder are pooled via sync.Pool (GetEncoder/PutEncoder, GetDecoder/PutDecoder) so Marshal/Unmarshal avoid a fresh allocation per call; changing that pooling contract or the central struct-field cache would ripple through every encode/decode path in the package.

Tech Stack Pure Go with two runtime dependencies declared in go.mod: github.com/vmihailenco/tagparser/v2 for parsing the msgpack struct tag syntax, and github.com/stretchr/testify as a test-only assertion library (go-spew, go-difflib, yaml.v3 come in transitively as testify’s own indirect deps). The module targets Go 1.19+, ships an extra/msgpappengine subpackage for legacy Appengine datastore.Key/datastore.Cursor support, and uses build tags (safe.go vs unsafe.go) to switch between a safe and an unsafe-pointer-based string/byte conversion depending on the appengine build tag. CI is GitHub Actions (build.yml, commitlint.yml, release.yml), with commitlint.config.js indicating releases are cut via conventional-commit-driven semantic-release tooling.

Code Quality Testing uses the standard testing package with testify’s require/assert, spread across msgpack_test.go, types_test.go, ext_test.go, and intern_test.go, plus godoc-style Example* functions (example_test.go, example_CustomEncoder_test.go, example_registerExt_test.go) that double as both documentation and executable tests, and a dedicated bench_test.go for performance regressions. Error handling favors typed sentinel errors (e.g. unexpectedCodeError) over generic fmt.Errorf strings on hot decode paths, and public API naming is consistent with encoding/json-style conventions. There is no visible linter config or static-analysis step wired into the CI workflows beyond go vet/build, and the project has been in low-activity maintenance mode for some time, with a moderate backlog of open issues relative to its release cadence.

What Makes It Unique Unlike code-generation-based Go MessagePack libraries, msgpack/v5 works purely through reflect with no go generate step, while still caching per-type encode/decode functions so repeated calls avoid re-deriving reflection logic. Its combination of a SetCustomStructTag escape hatch (letting existing json-tagged structs be reused as-is) and an extension-type registry for preserving Go type identity across the wire is a more complete customization surface than most same-purpose libraries offer, at the cost of the raw throughput a codegen-based encoder can reach.