smart_open

smart_open is a Python library that acts as a drop-in replacement for the built-in open() function, transparently streaming very large files to and from remote storage backends including S3, Google Cloud Storage, Azure Blob Storage, HDFS/WebHDFS, HTTP/HTTPS, and SSH/SCP/SFTP, in addition to the local filesystem. It falls back to native open() wherever possible, so existing code that iterates lines or reads/writes bytes generally keeps working when a local path is swapped for a s3://, gcs://, or azure:// URI.

Beyond unifying the read/write interface across storage backends, smart_open handles on-the-fly compression and decompression for .gz, .bz2, .xz, .zst, and .lz4 files based on the filename extension (with the algorithm and per-call options overridable), and exposes transport-specific parameters — such as passing a preconfigured boto3 client for S3 — so callers keep full control over authentication and connection tuning while avoiding the boilerplate normally required to wrap each cloud SDK’s native upload/download methods into a file-like object.

What You Get

• A drop-in open() replacement supporting s3://, gcs://, azure://, hdfs://, webhdfs://, http(s)://, and ssh|scp|sftp:// URI schemes alongside local paths
• Transparent on-the-fly compression/decompression for .gz, .bz2, .xz, .zst, and .lz4 based on file extension, with a register_compressor hook to add custom formats
• transport_params for passing a preconfigured client (e.g. a tuned boto3 S3 client) instead of relying on default credential discovery
• Support for S3 multipart uploads and single-part uploads with writebuffer control (e.g. tempfile.TemporaryFile to reduce memory footprint)
• Full context-manager and io.IOBase compatibility, including seek(), so streamed remote files behave like regular Python file objects

Common Use Cases

• Reading and writing large datasets stored in S3, GCS, or Azure Blob Storage from data pipelines without hand-rolling boto3/google-cloud-storage/azure-sdk file wrappers
• Streaming compressed log or corpus files (.gz, .bz2, .xz) directly from cloud storage without downloading and decompressing them to local disk first
• Processing files from HDFS/WebHDFS clusters or over SFTP using the same iteration and read/write code paths used for local files
• Building ML/NLP data-loading pipelines (a common use case originating from its Gensim-ecosystem roots) that stream training corpora too large to fit in memory

Under The Hood

Architecture - smart_open_lib.py parses the URI scheme of the path passed to open() and dispatches to the matching transport module (s3.py, gcs.py, azure.py, hdfs.py, http.py, ssh.py, webhdfs.py, local_file.py), each of which wraps the storage backend’s native client in a file-like object satisfying Python’s io interfaces; compression.py layers transparent (de)compression on top of whichever transport returned the raw byte stream, keyed by file extension via a registry (register_compressor). Tech Stack - pure Python (99%+), packaged with pyproject.toml, with each transport’s third-party dependency (boto3, google-cloud-storage, azure-storage-blob, paramiko, etc.) declared as an optional extra so the base install stays dependency-free; a smart_open[all] extra pulls in every transport’s dependencies at once. Code Quality - an 18-file tests/ suite plus a separate integration-tests/ directory exercising real cloud backends, doctest-style examples embedded directly in the README (doctools.py regenerates them), and a help.txt/help('smart_open') reference kept in sync with the API; the project is actively maintained with roughly 15 commits/month and 70 tagged releases. API Design - the entire library funnels through one function, smart_open.open(), matching the shape and semantics of the stdlib open() as closely as possible (same context-manager behavior, same iteration-over-lines default) so migrating existing file-handling code to remote storage is close to a one-line change.