cloudpathlib

cloudpathlib gives cloud object storage the exact same interface as Python’s standard pathlib.Path, so code that already walks, globs, reads, and writes local files can target s3://, gs://, or az:// URIs with almost no changes. It wraps boto3, google-cloud-storage, and azure-storage-blob behind a uniform CloudPath API, transparently caching remote files to a local temp directory (or a persistent cache folder you control) so reads and writes happen against real files on disk without extra bookkeeping in your code.

What You Get

• A CloudPath factory that auto-dispatches on URI scheme (s3://, gs://, az://) to the matching client and path subclass
• Full read/write support — write_text/write_bytes/.open('w') upload changes to cloud storage with no manual upload calls
• Seamless local caching: files download on first access and are reused across calls, with a configurable persistent cache directory shared across processes
• Local filesystem test doubles (LocalS3Path, LocalGSPath, LocalAzureBlobPath) that let you mock cloud storage in unit tests without hitting real infrastructure
• An extensible base (CloudPath + Client) for adding support for additional cloud storage backends beyond the three built-in providers

Common Use Cases

• Swapping local file I/O for cloud storage in data pipelines without rewriting path-handling logic
• Writing ML/data-science code that reads training data or writes artifacts to S3/GCS/Azure using familiar Path-style calls
• Unit testing cloud-storage-backed code against local mock implementations instead of live buckets
• Building CLI tools or scripts that need to transparently work across local paths and multiple cloud providers

Under The Hood

Architecture: cloudpathlib’s core is a CloudPathMeta metaclass (cloudpathlib/cloudpath.py) that dispatches CloudPath(uri) to the right concrete subclass by matching the URI scheme (s3://, gs://, az://, http(s)://) against registered provider implementations in cloudpathlib/s3/, cloudpathlib/gs/, cloudpathlib/azure/, and cloudpathlib/http/. Each provider pairs a *Path class with a *Client (defined in cloudpathlib/client.py’s abstract Client base) that wraps the provider SDK (boto3, google-cloud-storage, azure-storage-blob) and manages a local cache directory files are downloaded into on first access, with writes uploaded back on close. Tech Stack: pure Python 3.9+, dependency-free at the base (only typing-extensions on old versions), with provider SDKs installed via extras (cloudpathlib[s3,gs,azure]); packaged with flit_core. Code Quality: an extensive tests/ suite covers every path operation per-provider plus dedicated local mock clients (tests/mock_clients/) for S3, GCS, and ADLS Gen2, enforced by black, flake8, and mypy in CI (make lint). API Design: the deliberate one-to-one mirroring of pathlib.Path method names and semantics means anyone familiar with the standard library needs almost no new API surface to learn, and the extras-based install keeps unused cloud SDKs out of dependency trees.