HdfsCLI

HdfsCLI (the hdfs package) provides both a Python client and a command line interface for the Hadoop Distributed File System over the WebHDFS and HttpFS protocols. Its Client class exposes intuitive methods for listing, reading, writing, and managing files on secure or insecure clusters, while the hdfscli command adds file transfers and an interactive Python shell with namenode aliasing.

Beyond the core client, HdfsCLI ships optional extensions for reading and writing Avro files, Kerberos authentication, and pandas DataFrame integration, making it a practical bridge between Python data workflows and Hadoop storage.

What You Get

• A Python Client API for WebHDFS and HttpFS supporting secure and insecure clusters
• The hdfscli command line tool for file transfer and an interactive shell
• Namenode URL aliasing via a configuration file for convenient access
• Optional extensions for Avro, Kerberos authentication, and pandas DataFrames

Common Use Cases

• Reading and writing files on an HDFS cluster from Python data pipelines
• Transferring files to and from HDFS via the command line
• Streaming Avro datasets directly out of HDFS for processing

Under The Hood

Architecture - The core hdfs/client.py defines a Client that issues WebHDFS REST requests through a persistent requests.Session, with InsecureClient and TokenClient variants for auth; config.py handles alias resolution from an INI file, __main__.py powers the hdfscli entry point, and the ext package layers Avro, Kerberos, and DataFrame support on top. Tech Stack - Pure Python built on requests and docopt, with optional dependencies (fastavro, pandas, requests-kerberos) gating the extensions; packaged via setuptools with a console_scripts entry point. Code Quality - The project carries a real test suite (test/test_client.py, test_main.py, per-extension tests, plus fixtures under test/dat) and clear module separation, though recent development activity has slowed. API Design - The client API is compact and readable — client.list, client.read, client.write, client.status mirror HDFS operations directly — and the CLI’s alias mechanism removes repetitive namenode URLs, giving a low-friction developer experience.