object_store

object_store is a focused, async Rust crate that provides a single ObjectStore trait for interacting with object storage services. Instead of writing separate integration code for AWS S3, Google Cloud Storage, Azure Blob Storage, HTTP/WebDAV endpoints, and local or in-memory filesystems, applications write against one trait and swap backends with a runtime configuration change.

Originally developed by InfluxData and later donated to Apache Arrow, the crate is used in large-scale production systems such as crates.io and InfluxDB IOx. It emphasizes a small dependency footprint, atomic put/multipart-upload semantics, conditional reads and writes, vectored IO, and bulk deletion, making it a common lower-level building block for data-intensive Rust services and query engines like DataFusion and Delta Lake implementations.

What You Get

• A single ObjectStore trait implemented for AWS S3, Google Cloud Storage, Azure Blob Storage, HTTP/WebDAV, local filesystem, and in-memory backends
• URL-based store construction via parse_url/parse_url_opts, similar in spirit to fsspec, PyArrow FileSystem, and Hadoop FileSystem
• Atomic put, streaming get, multipart uploads, conditional (etag/version) reads and writes, vectored range reads, and bulk delete
• Composable adapters — LimitStore for bounding concurrent requests and ThrottleConfig for simulating rate-limited stores in tests
• A pluggable retry/backoff HTTP client layer shared across the cloud backends, with configurable crypto providers (ring or aws-lc-rs) and an optional bundled reqwest transport

Common Use Cases

• Building a Rust data engine or query layer (e.g. a DataFusion/Parquet/Delta Lake-style system) that must read and write the same dataset across S3, GCS, Azure, and local disk without backend-specific branching
• Swapping a service between local-disk development and cloud object storage in production purely through configuration, keeping the same read/write code path
• Implementing reliable, resumable large-file uploads to cloud storage via the crate’s multipart upload API
• Writing integration tests against an in-memory or local-filesystem ObjectStore implementation instead of mocking a specific cloud SDK

Under The Hood

Architecture - The crate centers on one core trait, ObjectStore (src/lib.rs), which every backend implements: LocalFileSystem (src/local.rs), InMemory (src/memory.rs), AmazonS3 (src/aws/), GoogleCloudStorage (src/gcp/), MicrosoftAzure (src/azure/), and HttpStore (src/http/). A companion ObjectStoreExt trait (src/lib.rs) layers convenience methods (put, get, put_multipart) over the lower-level _opts methods that backends actually implement, keeping the required trait surface small. Cloud backends share a common HTTP client stack under src/client/ — retry.rs and backoff.rs implement shared retry/backoff policy, builder.rs constructs configured clients, and s3.rs/get.rs/list.rs/pagination.rs hold protocol-shape helpers reused across the S3, GCS, and Azure implementations, which each speak their provider’s native REST API rather than going through a vendor SDK. Store selection from a URL happens through parse_url/parse_url_opts (src/parse.rs) and an optional ObjectStoreRegistry (src/registry.rs) that maps URL scheme/authority/path prefixes to already-constructed store instances. Cross-cutting adapters — LimitStore (src/limit.rs) for concurrency bounding and the ThrottleConfig wrapper (src/throttle.rs) for rate simulation — wrap any ObjectStore implementation transparently. Tech Stack - Rust 2024 edition (rust-version = 1.85), built on async-trait/native async fn in traits, bytes for zero-copy payloads, futures-core/futures-util for streaming, and url for parsing. Cloud backends are feature-gated: cloud-base pulls in serde/serde_json/quick-xml/chrono for XML/JSON protocol handling, with an optional bundled reqwest HTTP transport (feature reqwest) that can be swapped for a custom client on wasm32-wasip1 where reqwest doesn’t compile. TLS/crypto is pluggable via ring or aws-lc-rs feature flags rather than hard-wired. Errors use thiserror for a structured, matchable Error enum (src/lib.rs) rather than boxed dynamic errors at the API boundary. Code Quality - Unit tests are distributed across the crate’s own modules (59+ files containing #[test]/#[tokio::test], concentrated in lib.rs, local.rs, registry.rs, parse.rs, util.rs, config.rs, delimited.rs, payload.rs, attributes.rs, tags.rs, prefix.rs) plus integration tests under tests/ (http.rs, get_range_file.rs) and a generic src/integration.rs conformance suite that every backend runs against to verify consistent behavior. lib.rs opts into strict lint gates: #![deny(rustdoc::broken_intra_doc_links, ...)] and #![warn(missing_docs, missing_debug_implementations, clippy::future_not_send, unreachable_pub, ...)], and the repo runs clippy/rustfmt/typos checks in CI (.github/workflows/). API Design - The public surface is small and consistently named around one trait plus URL-based construction, and lib.rs carries extensive /// doc comments with runnable, feature-gated doctest examples (list, get, put, URL parsing) directly in the crate documentation, keeping the get-started path to a few lines of code against any backend.