rust-s3

rust-s3 is a Rust library for working with Amazon S3 or any S3-compatible object storage API, including Backblaze B2, Wasabi, Yandex Object Storage, MinIO, Cloudflare R2, and Google Cloud Storage. It exposes a single Bucket struct through which callers create, list, and delete buckets, and put, get, list, delete, tag, and head objects, with both path-style and subdomain-style URL support.

The library is built around the maybe-async crate, letting a single method body compile into tokio-async, async-std-async, or fully synchronous variants depending on which Cargo feature flags are enabled, plus a blocking feature that wraps the async paths in blocking wrappers via a proc macro. Requests are automatically retried once by default (configurable via set_retries), and the crate ships dedicated presign_get/presign_put/presign_post/presign_delete methods so short-lived signed URLs can be handed to third parties without making the bucket public.

It is organized as a Cargo workspace of three crates — rust-s3 (the s3 library crate), aws-region, and aws-creds — with AWS Signature V4 request signing implemented directly in signing.rs rather than delegated to the official AWS SDK, which keeps the dependency footprint and compile times low for services that only need S3-shaped object storage rather than the full AWS SDK surface.

What You Get

• A Bucket struct with create, delete, list_buckets, and exists operations, plus get_object, put_object, get_object_stream, put_object_stream, delete_object, and delete_objects for object-level work
• Presigned URL generation (presign_get, presign_put, presign_post, presign_delete) so uploads and downloads can be delegated to clients without exposing credentials or making the bucket public
• Three interchangeable backend implementations selected via Cargo features: with-tokio (reqwest), with-async-std (surf), and sync (attohttpc), each with native-tls and rustls-tls variants
• Object tagging support (put_object_tagging/get_object_tagging) and multipart upload handling via PutObjectStreamRequest, chunked at an 8 MiB CHUNK_SIZE
• Built-in AWS Signature V4 request signing (signing.rs) and credential/region handling split into companion aws-creds and aws-region crates
• A blocking feature that generates *_blocking variants of every async method for callers who don’t want to manage an async runtime

Common Use Cases

• Uploading and retrieving user-generated files (images, documents, backups) from a Rust backend service directly against S3 or a self-hosted MinIO cluster
• Serving object storage across multiple providers with one codebase — for example, using the same Bucket API against AWS in production and MinIO or LocalStack in local development
• Generating short-lived presigned upload/download URLs so a frontend or third party can read from or write to a specific object without holding AWS credentials
• Streaming large file uploads and downloads via the tokio or async-std backends instead of buffering entire objects in memory
• Running S3-compatible storage operations in synchronous Rust codebases (CLIs, scripts) via the sync feature, without adopting an async runtime

Under The Hood

Architecture The crate centers on a single Bucket struct (s3/src/bucket.rs, roughly 4,500 lines) that owns credentials behind an Arc<RwLock<Credentials>> and dispatches every operation through a Request trait implemented three times — ReqwestRequest for tokio, SurfRequest for async-std, and AttoRequest for the fully synchronous path (s3/src/request/). Method bodies are written once and adapted per backend using the maybe-async macro, so #[maybe_async::maybe_async] functions compile to async fn or plain fn depending on which Cargo feature is active, and the blocking feature layers a #[block_on] proc-macro wrapper on top to generate *_blocking variants without duplicating logic. Commands are represented as an enum in command.rs and turned into signed HTTP requests via signing.rs, which implements AWS Signature V4 directly rather than depending on the AWS SDK. This buys a small dependency graph at the cost of the crate owning its own signing correctness rather than inheriting AWS’s reference implementation.

Tech Stack Built on Rust 2024 edition as a three-crate Cargo workspace (s3, aws-region, aws-creds). The tokio backend depends on reqwest with streaming enabled; the async-std backend on surf; the sync backend on attohttpc. Serialization runs through serde/serde_json for JSON and quick-xml for S3’s XML responses, with minidom gated behind the tags feature for tag-document parsing. Errors are unified through a thiserror-derived S3Error enum that wraps each backend’s error type behind #[cfg(feature = ...)] variants. TLS is selectable per backend (native-tls vs rustls-tls), and time handles the RFC2822/ISO8601 date formatting AWS signing requires.

Code Quality Unit tests live inline alongside implementation code (#[cfg(test)] modules across roughly a dozen files in s3/src, including bucket.rs), and GitHub Actions runs make ci and make test-all on every push and pull request against master/dev, exercising multiple feature-flag combinations. Error handling is explicit throughout — the public API returns Result<_, S3Error> rather than panicking, and S3Error is #[non_exhaustive] so new variants don’t break downstream matches. The heavy use of #[cfg(feature = ...)] gating to support three backends from one codebase adds real complexity to bucket.rs, which has grown into a very large single file; there’s no separate integration-test directory visible in this shallow clone, so end-to-end coverage against a live or mocked S3 endpoint isn’t verifiable from source alone.

API Design The public surface is a single Bucket struct with consistently named methods (get_object/put_object/delete_object, each with _stream and, behind the blocking feature, _blocking siblings), which keeps the mental model small once a caller picks a backend feature. The tradeoff is upfront: consumers must choose the right combination of Cargo features (runtime + TLS backend, plus optional tags/blocking) before anything compiles, and the crate ships six runnable examples (tokio, async-std, sync, minio, r2, google-cloud) precisely because that feature selection isn’t obvious from the README alone. Presigned URL methods and multipart streaming uploads are available without extra setup beyond credentials and a region, which is a meaningfully lower-boilerplate entry point than the official AWS SDK for S3-only use cases.