avro-rs

apache-avro is the official Rust implementation of Apache Avro, the widely used binary data-serialization system built for compact storage and schema evolution. It gives Rust developers two ways to work with Avro data: a generic Value type for dynamic, schema-agnostic access, or Serde-derived Rust types for fully static, compile-time-checked serialization.

The crate covers the full Avro toolchain — schema parsing and validation, binary encoding/decoding, container-file reading and writing, schema resolution between writer and reader schemas, and optional compression codecs (Snappy, Zstandard, Bzip2, Xz). It’s maintained under the Apache Software Foundation as part of the broader Avro project, with active releases and a workspace that also ships a derive crate for generating schemas straight from Rust structs.

What You Get

• Binary encode/decode functions for the Avro data format, plus container-file Reader/Writer types
• A full Avro schema parser and validator supporting all primitive, complex, and logical types
• An optional derive feature for generating Avro schemas directly from annotated Rust structs
• Built-in compression codec support (Deflate, Snappy, Zstandard, Bzip2, Xz) selectable via Cargo features
• Schema resolution and compatibility checking between writer and reader schemas
• Single-object encoding support for Confluent-style Kafka wire format use cases

Common Use Cases

• Producing and consuming Avro-encoded messages in Kafka-based streaming pipelines
• Reading and writing Avro container files for batch data processing and archival
• Enforcing schema evolution rules when producer and consumer schemas drift apart
• Serializing typed Rust structs to Avro directly via Serde derive macros
• Interoperating with JVM-based Avro tooling (Java/Scala) that shares the same binary wire format

Under The Hood

Architecture The workspace splits into the avro crate (the actual apache-avro library), avro_derive (a proc-macro crate for #[derive(AvroSchema)]), avro_test_helper, and a wasm-demo. The library itself is organized as focused, single-responsibility modules re-exported from a thin top-level API: schema/ (parsing, builders, name resolution, record and union handling), types.rs (the dynamic Value enum), encode.rs/decode.rs (the low-level binary codec), reader/ and writer/ (container-file and datum-level Reader/Writer, plus single-object encoding), serde/ (the Serialize/Deserialize bridge and AvroSchema/AvroSchemaComponent traits), and schema_compatibility.rs/schema_equality.rs for evolution checks. Both the dynamic Value path and the static Serde path funnel through the same schema and encode/decode layers, so the Schema/Value representations are the crate’s one true breaking-change surface — everything else is built on top of them.

Tech Stack A pure Rust crate on edition 2024 with an MSRV of 1.88.0. Core dependencies include serde/serde_json for the static (de)serialization path, thiserror for structured errors, num-bigint/bigdecimal and uuid for Avro’s decimal and UUID logical types, digest for schema fingerprinting, and miniz_oxide for the always-on Deflate codec, with crc32fast/snap, zstd, bzip2, and liblzma gated behind optional Cargo features. ouroboros handles self-referential structs and bon supplies builder ergonomics. Target-specific dependencies split real rand (native) from quad-rand (wasm32), backed by a companion wasm-demo workspace member. CI runs a dedicated test workflow, a clippy workflow, cargo-audit, CodeQL, and zizmor for workflow security linting.

Code Quality Tests are extensive and colocated — a dedicated tests/ directory holds dozens of regression tests named after issue/PR numbers (avro-3786.rs, avro-rs-219.rs, and similar), alongside inline #[cfg(test)] mod tests blocks in core files that use pretty_assertions for readable diffs. Error handling is fully typed: a single Error struct wraps a boxed Details enum via thiserror, avoiding stringly-typed errors in favor of structured, matchable variants. Workspace-level clippy lints (clippy.all = "warn", plus doc_markdown and too_long_first_doc_paragraph) are enforced in CI through a dedicated clippy workflow. Naming follows consistent Rust convention, and public APIs are documented with doc comments kept in sync with the README via cargo-rdme. The combination of typed errors, workspace-wide lint enforcement, and a CI matrix spanning tests, clippy, cargo-audit, and CodeQL points to strong code-quality discipline.

API Design The public surface is deliberately narrow — lib.rs re-exports only the types callers actually need (Schema, Reader, Writer, Value, AvroSchema, to_value/from_value) while keeping internal modules private, so getting started means picking one of two clear entry points: the dynamic Value API or the Serde-derived static API. Optional functionality (compression codecs, the derive macro) is opt-in via Cargo features rather than always-on dependencies, keeping the default build lean. Documentation leans on a dedicated documentation module (primer, dynamic-vs-static guides, data-model mapping docs) rather than scattering guidance across the README alone, and the crate ships runnable examples covering single-object encoding, interop data generation, and benchmarking — lowering the ramp-up cost for a format with as much conceptual surface (schema evolution, logical types, wire formats) as Avro has.