streaming-decompressor

streaming-decompression provides Decompressor, an implementation of FallibleStreamingIterator purpose-built for decoding streams of compressed items without allocating a fresh buffer for every entry. It wraps an iterator of compressed items and, on each advance() call, decompresses the next entry into an internal Vec<u8> that is automatically recovered from the previous item and reused once the consumer is done with it.

The crate defines two small traits, Compressed and Decompressed, that let a caller describe how to detect whether an item actually needs decompressing and how to expose its output buffer, so the reuse logic works with any concrete compression scheme rather than one hard-coded codec. It is small and dependency-light (a single dependency on fallible-streaming-iterator) and originates from the author’s columnar-data crates, where it factors out a buffer-reuse pattern needed to stream-decompress successive pages of a file format efficiently.

What You Get

• A Decompressor<I, O, F, E, II> streaming iterator that wraps any Iterator<Item = Result<I, E>> of compressed items.
• Compressed and Decompressed traits for describing runtime-compression state and buffer access on your own item types.
• Automatic buffer hand-off between consecutive advance() calls, plus into_inner() to reclaim the buffer once you’re done iterating.
• Zero unsafe code (#![forbid(unsafe_code)]) and a single runtime dependency on fallible-streaming-iterator.

Common Use Cases

• Streaming decompression of columnar file-format pages, such as Parquet-style block decoding.
• Sequential decompression of a stream of compressed log or event records.
• Any allocation-sensitive Rust pipeline that decompresses many small items in a loop and wants to avoid per-item Vec allocation.

Under The Hood

Architecture The crate is a single file (src/lib.rs, roughly 110 lines of core logic plus tests) defining two traits (Compressed, Decompressed) and one generic struct, Decompressor<I, O, F, E, II>, implementing the FallibleStreamingIterator trait from the fallible-streaming-iterator crate. All the logic lives in advance(): it reclaims the buffer from the previously-decompressed item (via buffer_mut, using mem::take) when that item was actually decompressed, pulls the next item from the wrapped iterator, and calls the caller-supplied decompression closure with the reclaimed buffer, storing the result as current. get() and size_hint() are thin delegations. Because the struct holds nothing beyond a buffer, the current item, and a was_decompressed flag, there’s no separation-of-concerns problem to speak of — it is a minimal decorator over an existing iterator, and any change to the core abstraction (for example, supporting async streams) would touch essentially the whole crate since everything funnels through advance().

Tech Stack Pure Rust, edition 2018, with a single runtime dependency, fallible-streaming-iterator = "0.1", which is re-exported at the crate root so consumers don’t need to add it separately. There is no async runtime, no I/O, no database, and no application framework involved — this is a leaf utility crate meant to be embedded inside larger data-processing crates, consistent with its origin in the author’s arrow2/parquet2 columnar-data ecosystem. No CI workflow is committed to the repository.

Code Quality Two unit tests (test_basics_uncompressed, test_basics_compressed) cover the compressed and uncompressed buffer-reuse paths, but there is no integration-test suite and, since no CI config is checked in, the tests aren’t verified to run automatically on changes. Error handling is fully typed and explicit: the crate is generic over E: std::error::Error and propagates Result via ? throughout advance(), never swallowing errors. Naming is clear and idiomatic (Compressed, Decompressed, Decompressor, is_compressed, buffer_mut), unsafe code is forbidden crate-wide, and every public item carries a doc comment — but there is no committed rustfmt or clippy configuration to enforce style automatically.

API Design The public surface is intentionally small: one generic struct plus two traits, no macros, no unsafe blocks, and a closure-based Fn(I, &mut Vec<u8>) -> Result<O, E> signature that leaves the actual decompression algorithm entirely up to the caller rather than hard-coding one. That generality is also the main onboarding cost — the five type parameters on Decompressor (I, O, F, E, II) read as intimidating on first encounter, and the crate leans on a fully worked example embedded in its docs (via include_str! from lib.md) to make usage concrete. It doesn’t introduce a new compression algorithm or file format; it factors out a narrowly-scoped buffer-reuse pattern that the author needed repeatedly in his other columnar-data crates.