hf-xet

hf-xet is the Python package (Rust core with PyO3 bindings, built via maturin) that gives huggingface_hub access to Xet storage, Hugging Face’s chunk-based deduplication transport for the Hub. Instead of transferring and storing whole files, Xet splits models and datasets into content-defined chunks, deduplicates chunks shared across versions or files, and only transfers what has actually changed, with a local chunk-based disk cache layered alongside huggingface_hub’s existing cache.

The package is not designed to be used directly by application code; it is pulled in automatically as a dependency of huggingface_hub and activated transparently whenever a repository on the Hub is Xet-enabled, handling concurrent network communication with HF’s Content-Addressed Storage (CAS) backend behind the scenes. It ships as a compiled Rust extension for each supported platform, with a companion git-xet CLI (Git LFS-compatible) and WebAssembly-target support so downstream browser SDKs like hf-hub can also depend on the same core crates.

What You Get

• Chunk-based deduplication that skips retransferring and restoring content shared across files, models, and dataset versions
• A compiled Rust core (PyO3/maturin bindings) delivering high-throughput concurrent uploads and downloads to Hugging Face’s Xet CAS backend
• A local chunk-based disk cache that layers alongside huggingface_hub’s existing cache for faster repeat access
• Backwards compatibility with Git LFS semantics, plus a standalone git-xet CLI and WebAssembly build target for browser-based consumers

Common Use Cases

• Transparently accelerating huggingface_hub uploads and downloads of large model weights and dataset files without any user-facing configuration
• Reducing bandwidth and storage costs for teams repeatedly pushing incremental checkpoints of large models to the Hub
• Powering browser-based Hub integrations (e.g. hf-hub) that need the same chunking/dedup logic compiled to WebAssembly
• Git-LFS-compatible workflows via the git-xet CLI for repositories that mix traditional Git tooling with Xet storage

Under The Hood

Architecture - xet-core is a Rust workspace split into focused crates: xet_pkg (published as the hf-xet crate) exposes the high-level session API, xet_client handles HTTP communication with Xet CAS backend services, xet_data implements the chunking/deduplication/reconstruction pipeline, xet_core_structures defines shared types like MerkleHash and Xorb objects, and xet_runtime provides the async runtime, config, and logging infrastructure. The hf_xet/ directory wraps these crates with PyO3 bindings (py_file_download_handle.rs, py_upload_commit.rs, py_xet_session.rs, etc.) compiled via maturin into the hf-xet PyPI wheel that huggingface_hub imports.

Tech Stack - Overwhelmingly Rust (94.5% of the codebase), using pyo3 with abi3-py38 for stable-ABI Python bindings across versions 3.8+, maturin as the Python build backend, and a WebAssembly target (wasm32-unknown-unknown) so browser consumers can reuse the same crates via wasm-bindgen/tokio_with_wasm shims. A small Python test suite (hf_xet/tests/) exercises the compiled extension’s upload/download/session/progress APIs directly.

Code Quality - The workspace uses cargo test/cargo bench for the Rust core and cargo clippy -D warnings for linting, with CI enforcing a clean WebAssembly build in addition to native targets. The README goes further than most, documenting platform-specific diagnostic scripts (scripts/diag/) for collecting stack traces and core dumps when hf-xet misbehaves in production, and explicit patterns (conditional ?Send, web_time::Instant, filesystem gating) that wasm-reachable code must follow — evidence of a codebase actively guarding against regressions across its unusually wide target matrix.

API Design - hf-xet is explicit that it ‘is not meant to be used directly’ — its Python API exists purely to be called by huggingface_hub, so developer experience is optimized for that one integration point rather than general usability, with a narrow set of async session, upload, and download handle types instead of a broad public surface.