azure-data-lake-store-python

azure-datalake-store is Microsoft’s official pure-Python client for Azure Data Lake Storage Gen1 (ADLS Gen1). It wraps the ADLS WebHDFS-compatible REST API behind an AzureDLFileSystem object that exposes familiar filesystem verbs — ls, mkdir, touch, du, chmod, chown, glob matching, and ACL management — so code that already thinks in terms of paths and file handles can talk to ADLS with minimal changes.

Beyond basic file operations, the library provides Python file-like objects (open, read/write, context-manager support) that interoperate with libraries expecting standard file handles, such as reading a remote CSV directly into pandas. For bulk data movement it ships ADLUploader and ADLDownloader classes that chunk large files and parallelize transfer across multiple threads, plus a resumable state file so an interrupted transfer of a whole directory tree can pick back up rather than restart.

Authentication is handled via environment variables (tenant ID, client credentials) or programmatically through lib.auth(), with retry policies built in for transient REST failures. The project explicitly targets ADLS Gen1 only — Microsoft’s guidance for Gen2 is to use azure-storage-file-datalake instead, which matters when evaluating this package for a new project versus maintaining an existing ADLS Gen1 integration.

What You Get

• AzureDLFileSystem - a single object exposing ls, mkdir, touch, du, df, chmod, chown, glob, and ACL operations against an ADLS Gen1 account
• File-like objects - open() returns objects supporting read/write/tell/flush and context-manager usage, compatible with libraries like pandas that expect standard file handles
• ADLUploader / ADLDownloader - multi-threaded, chunked transfer classes for moving whole directory trees, glob-matched files, or single files with configurable thread count and buffer size
• Resumable transfers - upload/download state can be saved and reloaded so an interrupted directory transfer resumes instead of restarting
• Cross-platform path handling - AzureDLPath accepts string, pathlib.PurePath, and PureWindowsPath inputs and normalizes forward/backslash separators automatically
• Configurable retry policy - ExponentialRetryPolicy backs off and retries on 5xx, 401, 408, and 429 responses, with retry count and backoff factor as tunable parameters

Common Use Cases

• Bulk data ingestion pipelines - a data engineer scripts ADLUploader to push a local directory of generated files into ADLS Gen1 for downstream Hadoop/Spark processing
• Ad-hoc data science access - an analyst opens a remote file with adl.open() and streams it into pandas.read_csv() without downloading the whole file first
• Legacy ADLS Gen1 maintenance - a team supporting an existing ADLS Gen1 deployment keeps using this library while new projects are steered to azure-storage-file-datalake for Gen2
• Scripted account administration - an operator uses chmod/chown/get_acl_status to audit and adjust permissions across a large tree of files from a Python script instead of the Azure CLI

Under The Hood

Architecture azure-datalake-store separates concerns across five modules: lib.py implements the low-level DatalakeRESTInterface that wraps webHDFS-compatible REST calls and handles authentication token acquisition; core.py builds on top of that with the public-facing AzureDLFileSystem and AzureDLFile classes that expose filesystem semantics (ls, open, du, chmod) while delegating actual REST calls to the lib layer; multithread.py layers ADLUploader/ADLDownloader on top of AzureDLFileSystem for parallel chunked transfer, using get_chunk/put_chunk worker functions dispatched across a thread pool; retry.py isolates retry/backoff policy as a pluggable RetryPolicy class consumed by both lib.py and core.py; and exceptions.py centralizes typed exceptions. The layering is mostly clean, though core.py itself is dense, combining both AzureDLFileSystem and AzureDLFile in one large module, so a change to path handling or file-open semantics touches a large single file rather than an isolated component.

Tech Stack The library targets recent Python 3 releases and depends on requests for HTTP and optionally azure-identity for token-based auth via an “auth” extra. Packaging is classic setuptools (setup.py, no pyproject.toml or modern build backend), and the retry/backoff logic is implemented in-house rather than via an external library. There is no async support — everything is synchronous, with thread-based (not asyncio-based) parallelism used for bulk transfers via Python’s standard threading module.

Code Quality The tests/ directory holds a substantial pytest-based suite covering the filesystem, low-level REST, multithreaded transfer, and CLI layers, plus recorded HTTP cassettes and response mocking so REST interactions are tested against fixtures rather than live Azure calls. There are no static type annotations anywhere in the source and no linter or formatter configuration in the repo. Error handling mixes typed custom exceptions with Python-builtin equivalents shimmed for older interpreters.

API Design The library’s central design choice is to mirror a familiar filesystem/os-module vocabulary directly onto Azure’s ADLS Gen1 REST surface, and to return genuine Python file-like objects from open() so code written for local files, or libraries expecting a file handle, work with minimal adaptation. That is a pragmatic, low-friction API rather than a novel one — the library’s job is faithful, ergonomic wrapping of Azure’s own protocol, with the resumable, thread-parallel transfer state in ADLUploader/ADLDownloader standing out as the one place with genuinely non-trivial internal logic beyond plain REST wrapping.