hf_transfer

hf_transfer is a power-user library from Hugging Face that speeds up file transfers with the Hub on very high-bandwidth networks, where pure Python cannot saturate the available throughput. It ships a small Rust core, compiled to a native Python extension via pyo3 and maturin, that performs chunked, parallel HTTP downloads and uploads with retry and exponential backoff.

It exposes just two functions, download and upload, and is designed to be driven by huggingface_hub rather than used directly. When enabled via the HF_HUB_ENABLE_HF_TRANSFER environment variable, it lets model and dataset transfers exceed the ~500 MB/s ceiling typical of Python-only clients.

What You Get

• A native Rust extension that saturates high-bandwidth links beyond Python’s ~500 MB/s ceiling
• Chunked, parallel download and upload with configurable concurrency (max_files, chunk_size)
• Built-in retry logic with exponential backoff and jitter for flaky connections
• Drop-in acceleration for huggingface_hub via the HF_HUB_ENABLE_HF_TRANSFER environment variable

Common Use Cases

• Downloading large model weights or datasets from the Hub on fast networks
• Uploading large model checkpoints or dataset shards to a Hub repository
• Speeding up CI, training-cluster, and cloud-instance transfers where bandwidth is plentiful

Under The Hood

Architecture - The entire library is a single Rust module (src/lib.rs, ~491 lines) that exposes download and upload as pyo3 #[pyfunction]s under a #[pymodule]. Each call builds a multi-threaded Tokio runtime and blocks on an async routine: download_async/upload_async split the file into fixed-size chunks, spawn them onto a FuturesUnordered set bounded by a Semaphore (max_files open handles), and issue ranged HTTP requests via reqwest, seeking and writing chunks into the target file with tokio’s async file I/O. Failed chunks are retried with exponential_backoff (base wait, jitter, capped max).

Tech Stack - Rust 2021 with pyo3 0.26 (abi3-py38 extension module), tokio 1.42 (rt-multi-thread, fs), reqwest 0.12 with streaming, futures 0.3, tokio-util codecs, rand for jitter, and vendored OpenSSL. It is built and packaged as a Python wheel with maturin, requiring Python 3.7+.

Code Quality - The code is compact and idiomatic, with explicit argument validation (e.g. parallel_failures cannot exceed max_files) and centralized retry/backoff. There is no visible unit test suite in the repository; correctness is exercised indirectly through huggingface_hub. Error handling maps failures into Python exceptions and cleans up partial files on download failure.

API Design - The public surface is intentionally tiny: two functions with clearly documented tuning parameters (max_files, chunk_size, parallel_failures, max_retries, headers, callback). This keeps it predictable for power users, though it deliberately omits progress bars and general-purpose ergonomics, expecting callers to drive it through huggingface_hub rather than directly.