Datasets

Datasets is Hugging Face’s library for accessing and processing machine learning datasets, providing one-line dataloaders for thousands of public datasets hosted on the Hugging Face Hub alongside efficient tools for working with local CSV, JSON, Parquet, image, audio, video, and text data.

Built on Apache Arrow, it uses zero-copy memory-mapped storage so datasets can exceed available RAM without slowing down access, and it supports streaming mode so huge datasets can be iterated over without a full download. Smart caching avoids reprocessing the same transformation twice, and native converters interoperate with NumPy, Pandas, Polars, PyTorch, TensorFlow, JAX, and Spark.

What You Get

• One-line access to datasets hosted on the Hugging Face Hub across text, image, audio, video, and 3D medical formats
• Zero-copy Apache Arrow backend so datasets larger than RAM can still be processed efficiently
• Streaming mode to iterate over massive datasets without downloading them first
• Built-in map()/filter() transformations with multi-processing support via num_proc
• Native conversion to and from NumPy, Pandas, Polars, PyTorch, TensorFlow, JAX, and Spark
• Support for loading local files in CSV, JSON, JSONL, Parquet, HDF5, XML, text, image, and audio formats

Common Use Cases

• Loading a public benchmark dataset (e.g. SQuAD, GLUE) for fine-tuning or evaluating a model
• Streaming a dataset too large to fit on disk directly into a training loop
• Preprocessing raw text, image, or audio data with map() before feeding it into a PyTorch or TensorFlow pipeline
• Converting between Pandas/Polars DataFrames and Arrow-backed datasets for exploratory analysis
• Building and sharing a custom dataset on the Hugging Face Hub for reuse by other practitioners

Under The Hood

Architecture - The core of Datasets is the Dataset/DatasetDict object defined in src/datasets/arrow_dataset.py (7,400+ lines), a thin, memory-mapped wrapper over Apache Arrow tables. load.py orchestrates dataset resolution — checking the Hugging Face Hub, a packaged_modules loader (CSV/JSON/Parquet/etc.), or a user-supplied loading script — and hands off to builder.py, which downloads, verifies, and writes Arrow files via arrow_writer.py/arrow_reader.py. iterable_dataset.py provides the streaming code path that avoids materializing data on disk, and distributed.py/parallel/ handle multi-worker and multi-process execution.

Tech Stack - Written in Python (>=3.10), with pyarrow as the storage backbone, huggingface_hub for Hub access, fsspec for filesystem abstraction, and optional integrations with NumPy, Pandas, Polars, PyTorch, TensorFlow, and JAX. Packaging is managed with a legacy setup.py plus pyproject.toml for tool configuration (ruff, pytest), and CI runs via GitHub Actions.

Code Quality - The tests/ directory contains 1,300+ test functions covering the Arrow dataset core, builders, download manager, distributed execution, and each packaged data format, run under pytest with strict FutureWarning filtering on huggingface_hub. Ruff enforces linting (import sorting, unused imports) across the codebase. Naming is consistent and the public surface is concentrated in a small number of well-documented entry points (load_dataset, Dataset.map, Dataset.filter).

API Design - The library is built around a single memorable entry point, load_dataset(name, **kwargs), that returns a ready-to-use object regardless of whether the source is a Hub dataset, a local file, or a streaming source — minimizing the boilerplate needed to get from zero to a usable dataset. Chainable methods like .map(), .filter(), and .with_format() follow a consistent, discoverable naming convention across the whole API.