avro

hamba/avro is a Go implementation of the Apache Avro specification built for high-throughput serialization workloads. It compiles Avro schemas into cached, type-specific encoder/decoder pairs using unsafe pointers and reflect2 instead of the standard library’s reflect package, which keeps allocations and CPU overhead low even under heavy marshal/unmarshal traffic. Struct tags (avro:"field") drive the mapping between Go types and Avro fields, with built-in conversions for dates, durations, decimals, UUIDs, and other Avro logical types.

Beyond the core codec, the module ships companion packages for the parts of an Avro pipeline that most codecs leave out: ocf reads and writes Avro Object Container Files with Deflate, Snappy, ZStandard, or no compression; registry is a Confluent Schema Registry compliant HTTP client for fetching, registering, and checking compatibility of schemas; and soe implements Single Object Encoding for message-level framing. A gen/avrogen code generator can produce Go structs directly from .avsc schema files, and an avrosv CLI validates schemas in CI pipelines.

What You Get

• A Marshal/Unmarshal and streaming Encoder/Decoder API driven by parsed Avro schemas, with avro:"..." struct tags controlling field mapping
• Automatic Go type conversions for Avro logical types: dates and timestamps to time.Time, durations to time.Duration, decimals to *big.Rat, and UUID strings
• An ocf subpackage implementing Avro Object Container File reading/writing with Null, Deflate, Snappy, and ZStandard codec support
• A registry subpackage providing a Confluent Schema Registry HTTP client — schema fetch/create, subject/version listing, and compatibility checks
• A soe subpackage for Avro Single Object Encoding, plus flexible union handling via map[string]any, nullable pointers, or typed UnionConverter structs
• A code generator (gen package and avrogen CLI) that emits Go structs from .avsc schema files, and an avrosv CLI for schema validation in CI

Common Use Cases

• Encoding and decoding Kafka message payloads that carry Avro-serialized records
• Reading and writing Avro Object Container Files for data interchange or archival
• Validating producer/consumer schema compatibility against a Confluent Schema Registry before deploying a service
• Generating typed Go structs from a team’s canonical .avsc schema definitions instead of hand-writing them
• Building high-throughput data pipelines where reflection-based JSON/Avro codecs become a CPU bottleneck

Under The Hood

Architecture The library centers on a frozenConfig (created via Config{}.Freeze()) that owns pooled Reader/Writer instances and two sync.Map caches keyed by schema fingerprint plus Go rtype, so a given (schema, type) pair only pays the cost of building its encoder/decoder tree once, in codec.go’s decoderOfType/encoderOfType dispatch over the parsed Schema AST (schema.go). Record fields are wired up through a deferDecoder/deferEncoder indirection specifically to support self-referential and mutually recursive record schemas, and Ref schema nodes reuse the cache to short-circuit repeated resolution. Companion packages (ocf, registry, soe) are built as separate importable units on top of this core rather than folded into it, so a consumer that only needs, say, the registry client does not pull in the OCF block-compression codecs, and ocf.Decoder/ocf.Encoder compose the core avro.Reader/avro.Writer and a pluggable Codec interface for the container-file compression layer.

Tech Stack Written in Go (module targets Go 1.24, tested against 1.24 and 1.25 in CI) with a deliberately small dependency surface: modern-go/reflect2 for low-overhead reflection, json-iterator/go for the registry client’s JSON handling, klauspost/compress (zstd) and golang/snappy for OCF block compression, go-viper/mapstructure and ettle/strcase supporting the struct-tag/name mapping and code generator, and stretchr/testify plus golang.org/x/tools for testing and generation tooling. There is no external runtime beyond the standard library’s net/http for the registry client.

Code Quality The repository is extensively tested — roughly half of its Go files are _test.go files, including both black-box (avro_test package) and white-box internal tests (*_internal_test.go) that exercise cache and codec edge cases directly, plus dedicated example_test.go files per package that double as executable documentation. Tests use testify’s assert/require for readable failure output, and CI (.github/workflows/test.yml) runs golangci-lint (v2.6.2, config in .golangci.yml) and gotestsum across a two-version Go matrix on every push and pull request, so lint and test regressions are caught automatically.

What Makes It Unique Most Go Avro libraries stop at the codec; this one bundles the surrounding pipeline — Object Container Files, a Confluent Schema Registry client, Single Object Encoding, and a schema-to-struct code generator — as first-class, independently importable subpackages under one module. Its union handling is also unusually flexible for a static-typed language, accepting map[string]any, nullable pointers for two-branch nullable unions, or a typed UnionConverter interface for exhaustive compile-time-checked unions, letting callers pick the ergonomics-versus-safety tradeoff that fits their schema.