fastavro

fastavro is a Python implementation of the Apache Avro binary serialization format, purpose-built to close the performance gap left by the pure-Python avro reference package. Where the reference implementation can take over ten seconds to iterate a 10,000-record file, fastavro compiles its hot paths (_read, _write, _schema, _validation, and logical-type conversion) to C via Cython, bringing iteration time down to roughly what the official Java Avro SDK achieves — with a pure-Python fallback (_read_py, _write_py, etc.) that keeps the library usable on PyPy and other CPython-incompatible runtimes.

The library covers the full practical surface of Avro: block and record-level file readers and writers, schemaless (headerless) reading/writing for streaming use cases, JSON-encoded Avro I/O, schema resolution and aliasing, logical types (dates, decimals, UUIDs, and more), canonical-form schema parsing, and schema fingerprinting. Codec support spans Snappy, Deflate, Zstandard, Bzip2, LZ4, and XZ through optional extras. A bundled fastavro CLI lets you inspect .avro files from the shell without writing any code.

Avro is the de facto interchange format for Kafka topics, data-lake table formats, and cross-language schema-evolution pipelines, and fastavro is the library most Python services reach for when they need to read or write it without paying the reference implementation’s performance tax.

What You Get

• Cython-accelerated binary reader and writer (reader/writer) for full Avro Object Container Files, with an automatic pure-Python fallback when the compiled extension isn’t available
• Schemaless reader/writer for headerless Avro payloads, the pattern used by Kafka + Confluent Schema Registry integrations
• JSON Avro encoding via json_reader/json_writer for human-readable or debug-friendly output
• Schema tooling: parse_schema, canonical-form normalization, schema fingerprinting, and named-schema aliasing/resolution across versions
• Logical type support (date, time, timestamp, decimal, UUID, duration) with a pluggable reader/writer registry
• Optional codec extras (Snappy, Zstandard, LZ4, Deflate, Bzip2, XZ) installed via pip install fastavro[codecs]
• A fastavro command-line tool for inspecting Avro files without writing a script

Common Use Cases

• Producing and consuming Avro-encoded Kafka messages in Python services that sit alongside JVM-based consumers/producers
• Reading and writing Avro-formatted files in data-lake and ETL pipelines (Spark, data-lake table formats, batch exports)
• Validating and evolving record schemas across service versions using Avro’s schema-resolution rules
• Converting between Avro binary and JSON representations for debugging, logging, or interop with systems that only accept JSON
• Building lightweight command-line tooling to inspect or spot-check .avro files during data pipeline development

Under The Hood

Architecture fastavro is organized as parallel Cython/pure-Python implementation pairs: _read.pyx/_read_py.py, _write.pyx/_write_py.py, _schema.pyx/_schema_py.py, _validation.pyx/_validation_py.py, and _logical_readers.pyx/_logical_writers.pyx with their _py counterparts. The public modules (read.py, write.py) try to import the compiled _read/_write extensions first and transparently fall back to the _py modules (used automatically on PyPy, where Cython extensions aren’t built) — so callers get one stable API regardless of which backend loaded. Encoding/decoding primitives live under fastavro/io/ (binary_decoder.py, binary_encoder.py, json_decoder.py, json_encoder.py, plus a parser.py/symbols.py pair implementing Avro’s schema-resolution parsing rules), and fastavro/repository/ provides a pluggable schema-repository abstraction (base.py, flat_dict.py) for resolving named schemas. This split keeps the performance-critical byte-level codec logic isolated from the schema/validation layer that has to stay correct and hand-auditable.

Tech Stack The project is pure Python 3.10+ plus Cython for the compiled extensions (built via setup.py with Extension entries for _read, _schema, _write, _validation, _logical_readers, _logical_writers, each falling back to a checked-in .c file when Cython itself isn’t present at build time). Optional codec support is delegated to cramjam (Snappy/Zstandard bindings), backports.zstd, and lz4 via extras_require groups (codecs, snappy, zstandard, lz4) rather than hard dependencies, keeping the base install lightweight. Packaging uses a standard setuptools + pyproject.toml build-system declaration, and the library ships .pyi type stubs plus a py.typed marker for full static-typing support in consumers.

Code Quality The tests/ directory contains 26 Python test modules (test_fastavro.py, test_json.py, test_compression.py, test_aliases.py, test_canonical_form.py, test_fingerprint.py, and more) plus a large corpus of real .avro and schema fixture files, run via run-tests.sh/tox.ini across Python 3.10 through 3.14 and PyPy3 in CI. GitHub Actions workflows build and test on Linux, macOS, and Windows separately, plus a dedicated formatting-check workflow; mypy.ini configures type checking against the codebase (with ignore_missing_imports carve-outs for optional codec deps lacking stubs). The project ships .pyi stub files alongside every Cython/pure-Python module pair, giving downstream type checkers accurate signatures regardless of which backend is active at runtime.

What Makes It Unique fastavro’s defining technical choice is maintaining a dual-backend architecture — Cython-compiled and pure-Python — behind one import surface, rather than requiring the compiled extension or dropping PyPy support outright. This lets the library serve both throughput-sensitive CPython deployments and PyPy users (where Cython C-extensions typically don’t apply) from a single codebase and API, without forcing users to choose between speed and portability. The project explicitly documents itself as being in maintenance mode, prioritizing correctness and compatibility fixes over new feature development.