encoding

encoding is a Go package from Segment collecting efficient encoders and decoders for the data formats a high-throughput messaging pipeline touches most often. Its flagship subpackage, json, is a byte-for-byte API-compatible replacement for the standard library’s encoding/json — adopting it is a single import-path change — that cuts CPU time and heap allocations dramatically by avoiding reflection on hot paths and using unsafe-pointer-based codec generation instead.

Alongside json, the module ships ascii (allocation-free ASCII validation and case folding), iso8601 (fast date-string validation without the allocation cost of time.Parse), proto (a reflection-light Protobuf wire-format encoder/decoder), and thrift (binary and compact Thrift protocol implementations). Each subpackage is independently importable and has no dependencies outside Go’s runtime and Segment’s own asm module, making it easy to adopt just the piece a project needs.

What You Get

• A json subpackage whose exported API mirrors encoding/json exactly, so switching is a one-line import change with no code rewrite required.
• Documented benchmark comparisons against both the standard library and github.com/json-iterator/go showing large reductions in time, allocations, and bytes/op.
• An ascii subpackage for allocation-free ASCII validity checks and case-insensitive comparisons used internally by the JSON encoder’s fast paths.
• An iso8601 subpackage for validating date-like strings without the heap-allocation cost of a failed time.Parse call.
• Reflection-avoiding proto and thrift subpackages that implement Protobuf and Thrift wire encoding directly against Go struct memory layout.
• A GitHub Actions benchmark workflow that tracks performance regressions on every change, in addition to the standard test suite.

Common Use Cases

• Swapping encoding/json for github.com/segmentio/encoding/json in services that serialize or deserialize large volumes of JSON on a hot path, to cut CPU and GC pressure without changing calling code.
• Validating incoming string fields as ISO8601 dates cheaply before deciding whether to parse them as time.Time or keep them as plain strings.
• Encoding or decoding Protobuf or Thrift payloads in performance-sensitive services without depending on protoc-generated code paths that rely on reflection.
• Building data-pipeline or event-ingestion systems (in Segment’s own case, marshaling/unmarshaling messages for queuing and storage) where allocation counts directly affect infrastructure cost.

Under The Hood

Architecture The module is a thin umbrella around independently-usable, format-specific subpackages — ascii, iso8601, json, proto, and thrift — plus an internal/runtime_reflect helper shared across them. There’s no central dispatcher: each subpackage is a self-contained alternative to a stdlib or common third-party serializer, built around the same shape of codec: a per-type encodeFunc/decodeFunc pair generated once via unsafe-pointer-based reflection and cached in an eventually-consistent, atomically-updated map (see json/codec.go), so encountering a new Go type triggers building just that type’s codec rather than recompiling the package. Because each subpackage depends only on internal/runtime_reflect and the external github.com/segmentio/asm module, a change to one format (say, ascii’s fast-path string escaping) is isolated from proto or thrift.

Tech Stack go.mod declares Go 1.23 with a single direct external dependency, github.com/segmentio/asm (SIMD-accelerated primitives, also maintained by Segment), plus an indirect golang.org/x/sys. There’s no web framework, ORM, or database — this is an embeddable library installed via go get and consumed as an import-path swap for encoding/json, google.golang.org/protobuf, or Thrift codegen. CI runs through GitHub Actions (test.yml for the test suite, benchmark.yml for performance regression tracking), and a Makefile drives local test/benchmark runs; fuzz tests under json/fuzz are pulled in only via go mod tidy and aren’t a runtime dependency.

Code Quality The repo carries roughly 32 _test.go files, and notably the json subpackage forks and adapts a large slice of the Go standard library’s own encoding/json test suite (golang_decode_test.go, golang_encode_test.go, golang_scanner_test.go, golang_tagkey_test.go, golang_bench_test.go) to assert wire-compatible behavior against the stdlib, on top of package-specific tests and a dedicated json/bugs regression directory plus a fuzz corpus. proto and thrift each carry their own decode/encode/struct/reflect test files covering per-type wire behavior. Errors are explicit and typed (dedicated error.go files in proto and thrift) rather than panicking. No local linter configuration was found, though the README displays a Go Report Card badge suggesting external static-analysis monitoring, and CI runs the full test suite on every push.

API Design The defining design choice is exact API compatibility with the standard library: the json subpackage’s Marshal, Unmarshal, Encoder, Decoder, and Marshaler/Unmarshaler surface mirror encoding/json precisely, so adoption requires only an import-path change — a deliberate contrast, called out directly in the README, against other fast-JSON libraries that require code generation or a different API shape. The proto and thrift subpackages follow the same drop-in philosophy for their formats. The trade-off, acknowledged directly in the README, is that internals lean on unsafe.Pointer arithmetic and reflection-avoidance techniques that make the codebase itself less approachable to contributors, in exchange for a near-zero-friction upgrade path for consumers.