opendal

Apache OpenDAL is a Rust data access layer that gives applications a single Operator API for interacting with dozens of storage backends — S3, GCS, Azure Blob, HDFS, WebDAV, Redis, PostgreSQL, and more — without hand-rolling a client for each service. Built around a zero-cost core, it composes optional layers for retries, timeouts, logging, tracing, metrics, and traffic control on top of any backend, so production concerns don’t leak into application code.

The project extends beyond Rust through official bindings for Python, Node.js, Java, Go, C, C++, and more, letting teams reuse the same storage abstraction across a polyglot stack. As an Apache Software Foundation project, OpenDAL is maintained by a broad open community and used by data infrastructure projects that need to swap storage backends without rewriting application logic.

What You Get

• A single Operator API across 60+ storage services (S3, GCS, Azure Blob/Data Lake, HDFS, WebDAV, FTP, Redis, PostgreSQL, and more)
• Composable layers for retry, timeout, logging, tracing, metrics, and concurrency control that wrap any backend without touching business logic
• Official language bindings for Python, Node.js, Java, Go, C, C++, and more, sharing one underlying implementation
• A pluggable HTTP transport layer (reqwest by default) so applications can swap TLS stacks or bring their own transport
• A shared behavior test suite that runs identical correctness tests against every backend implementation

Common Use Cases

• Building storage-agnostic applications that need to support multiple cloud providers without vendor lock-in
• Adding production-grade retry, timeout, and observability behavior to existing storage access code via layers
• Powering data lake and lakehouse engines that read and write across S3, GCS, HDFS, and other backends through one interface
• Migrating between storage providers without rewriting application-level storage code
• Giving polyglot services (Rust, Python, Node.js, Java) a consistent storage API via shared bindings

Under The Hood

Architecture OpenDAL’s core organizes around three abstractions: raw::Accessor (the low-level per-service trait implementing operations like stat/read/write/list/delete), raw::Layer (composable middleware wrapping the service stack, defined in raw/layer.rs), and the public Operator (types/operator) that end users call. Each storage backend lives under services/<name> implementing Accessor against that backend’s native API, while cross-cutting concerns (retry, timeout, logging, tracing, metrics, concurrency limiting) are implemented as separate Layer crates under layers/*, each pulled in optionally via Cargo features and composed onto the service stack through apply_service/apply_context hooks. The workspace is a monorepo of dozens of crates (core, dozens of services/* adapters, layers/* decorators, http-transports/*, and language bindings) unified by a facade crate that re-exports opendal-core and registers enabled services and transports through auto-register-services. This is a ports-and-adapters design: Operator is the port, each services/* crate is an adapter, and layers are decorators — changing the core Accessor trait would ripple through every service adapter, which is why it’s kept in the semver-exempt raw:: module rather than the public API.

Tech Stack Rust 2024 edition with a minimum rust-version of 1.91. The async runtime is tokio, feature-gated behind executors-tokio/internal-tokio-rt so consumers only pull it in when needed. HTTP transport is pluggable — the default is reqwest with rustls (aws-lc-rs) or native-tls, with a separate opendal-http-transport-reqwest crate for alternate configurations. Core dependencies include bytes, http, futures, serde/serde_json, uuid, url, quick-xml (for S3-style XML APIs), log plus logforth for structured logging, and reqsign-core for cloud request signing (S3/GCS/Azure). The Cargo workspace exposes granular per-service and per-layer feature flags (services-s3, layers-retry, and so on) so consumers only compile the backends they use, and docs.rs metadata sets all-features = true for full API documentation. CI runs an extensive set of GitHub Actions workflows covering per-service integration tests, cross-language bindings, and release automation.

Code Quality The project maintains a dedicated behavior test suite (opendal-testkit) shared across every service implementation, so all backends are exercised against the same correctness test matrix rather than one-off per-service tests — a strong pattern for a multi-backend abstraction library. Error handling is explicit and typed via a dedicated Error/ErrorKind enum with #[non_exhaustive] categories (NotFound, PermissionDenied, and so on) and backtrace capture, rather than stringly-typed errors. Naming is consistent across the workspace (the services-<name>/layers-<name> feature convention), and #![deny(missing_docs)] is set at the crate root to enforce documentation coverage on all public items. Clippy lints are configured per-crate. CI includes fuzzing and per-service correctness suites; some services rely on live-credential integration tests typical of multi-cloud storage libraries, so certain backends are exercised primarily through opt-in CI jobs rather than a default cargo test.

API Design The Operator API centers on ergonomic, method-based async calls (read, write, stat, list, delete) that stay identical regardless of backend, so switching from S3 to a local filesystem is a configuration change rather than a rewrite. Service configuration follows a builder/config-struct pattern with URI-based construction for quick setup, and layers compose through a fluent .layer(...) chain on Operator, letting consumers opt into retry, timeout, or logging incrementally. Getting-started and per-service examples ship under core/examples, and the crate’s top-level doc comment is generated directly from the README via include_str!, keeping docs.rs and the repository README in sync. The main DX cost is surface complexity: dozens of granular feature flags mean first-time users need to learn which services and layers to enable, though sensible defaults ease the common case.