snappy

snappy is a Node.js binding to Google’s Snappy compression algorithm, implemented in Rust and exposed via napi-rs instead of a pure-JS port or a hand-written C++ addon. It ships prebuilt binaries for 18 targets — every major desktop and server platform plus Android, FreeBSD, OpenHarmony, and a wasm32-wasi build for the browser — so consumers install it with zero native toolchain requirements in the common case.

The package exposes two distinct wire formats behind one API surface: one-shot raw Snappy blocks (compressSync/compress/uncompressSync/uncompress) for buffering entire payloads in memory, and the framed Snappy stream format (.sz) for incremental data via a Compressor/Decompressor class pair, WHATWG ReadableStream transforms (compressStream/uncompressStream), and ready-to-pipe Node Duplex factories (createCompressStream/createUncompressStream). Async one-shot calls run off the JS main thread using napi’s AsyncTask, and streaming decompression runs on a dedicated worker thread that stays deadlock-free under backpressure, making the library suitable for both quick request/response payloads and long-running pipe chains.

What You Get

• One-shot sync and async compression/decompression of raw Snappy blocks, accepting strings, Buffers, ArrayBuffers, or Uint8Arrays
• A Compressor/Decompressor class pair for incremental framed-Snappy (.sz) streaming with byte-identical output regardless of how input is chunked
• WHATWG ReadableStream transform functions (compressStream/uncompressStream) plus Node Duplex factory helpers for direct .pipe() chains
• Prebuilt native binaries for 18 Rust targets (Windows/macOS/Linux/Android/FreeBSD/OpenHarmony/wasm32-wasi) with automatic per-platform resolution, no build toolchain required
• Electron-safe output modes via copyOutputData, avoiding external buffer creation for V8 memory-cage compatibility on Electron >= 21
• AbortSignal support on the async one-shot APIs for cancelling in-flight compress/decompress calls

Common Use Cases

• Compressing API responses or cache payloads before storing them in Redis or a CDN to cut bandwidth and storage cost
• Building an ingestion pipeline that streams large log or event files through compressStream/createCompressStream without buffering the whole file in memory
• Producing or consuming .sz framed-Snappy files for interop with other Snappy-frame-format tooling (e.g. data pipelines that already emit framed Snappy)
• Running compression in Electron apps where the V8 memory-cage restricts external buffers, using the copyOutputData option for compatibility

Under The Hood

Architecture The crate cleanly separates three concerns across src/lib.rs, src/stream.rs, and src/stream_web.rs: one-shot block compression (sync functions plus Enc/Dec structs implementing napi’s ScopedTask for off-thread async execution), incremental framed streaming (the Compressor/Decompressor classes, which run decompression on a dedicated worker thread to stay deadlock-free under backpressure), and WHATWG Web Streams transforms built on tokio via napi’s web_stream feature. The wasm build is cfg-gated to drop stream_web entirely (no tokio runtime there) and falls back to a buffered class-API polyfill in the JS wrapper, so the same public API degrades gracefully across native and wasm targets without duplicating call sites.

Tech Stack The native layer is Rust on napi/napi-derive 3 (napi5 + serde-json + web_stream features) wrapping the snap crate for the actual Snappy codec, with mimalloc-safe set as the global allocator on non-wasm, non-Linux-musl/ohos targets for faster allocation. Builds are produced via the napi-rs CLI (napi build) targeting 18 platform triples including wasm32-wasip1-threads, with LTO and single codegen unit enabled in release profile. The JS-facing layer is TypeScript-typed through auto-generated .d.ts files, tested with ava using @oxc-node/core for on-the-fly TS execution, and linted/formatted via oxlint, cargo fmt, and taplo.

Code Quality Four spec files under __test__/ (index.spec.ts, streaming-class.spec.ts, streaming-web.spec.ts, node-stream-factory.spec.ts) cover the one-shot API, class-based streaming, Web Streams transforms, and Duplex factories respectively, run through ava with TypeScript support. Rust errors are explicitly typed and mapped through napi’s Error/Status rather than panicking, and both finish() calls on the streaming classes are idempotency-guarded against double-invocation. Husky plus lint-staged run formatting and linting pre-commit, and a CI workflow (.github/workflows/CI.yaml) exercises the matrix on every push.

API Design The library deliberately keeps two incompatible wire formats (raw one-shot blocks vs. framed streaming) behind clearly named, non-interchangeable functions rather than silently allowing cross-format misuse, and documents the incompatibility directly in the generated .d.ts comments. Ergonomics lean on idiomatic JS patterns — Promise-returning async variants, AbortSignal support, Duplex factories for drop-in .pipe() usage — while advanced options like copyOutputData are opt-in and documented with the specific compatibility scenario (Electron’s V8 memory cage) that motivates them, rather than being exposed as unexplained flags.