h5py

h5py is the standard Python interface to HDF5, the hierarchical binary format widely used in scientific computing, machine learning, and instrument data pipelines for storing huge multidimensional arrays alongside their metadata. It exposes HDF5’s file/group/dataset/attribute object model through ordinary Python objects: a File or Group behaves like a nested dictionary, and a Dataset supports NumPy-style slicing, fancy indexing, and broadcasting directly against data that may be far larger than available memory, since HDF5 only reads the requested hyperslab off disk.

Under the hood, h5py wraps the full HDF5 C API in a low-level Cython layer (one module per HDF5 API section — h5f, h5d, h5g, h5t, h5s, h5a, and more) and layers a high-level, NumPy-native Python API on top of it in h5py._hl. This gives users access to advanced HDF5 capabilities — chunked storage and compression filters, single-writer/multiple-reader (SWMR) concurrent access, virtual datasets that stitch multiple files into one logical dataset, and MPI-parallel I/O via mpi4py — without leaving Python or writing any C.

Because HDF5 is a single self-describing binary file that any HDF5-aware tool (MATLAB, Julia, C/Fortran codes, command-line utilities like h5dump) can also read, h5py is frequently the bridge between a Python/NumPy analysis pipeline and data produced or consumed by non-Python scientific tooling.

What You Get

• NumPy-style slicing and fancy indexing directly against on-disk HDF5 datasets, without loading the full array into memory
• A dict-like File/Group API for HDF5’s hierarchical namespace, plus an .attrs mapping for per-object metadata
• Chunked storage with built-in and third-party compression filters (gzip, LZF, szip, and pluggable custom filters)
• Single-writer/multiple-reader (SWMR) mode for concurrent read access while a file is actively being written
• Virtual datasets (VDS) that present many separate HDF5 files as one logical array
• Optional MPI-parallel I/O via mpi4py for multi-process reads and writes to a shared file

Common Use Cases

• Storing and incrementally appending large simulation or instrument output (detector frames, sensor time series) that exceeds available RAM
• Sharing multidimensional array data with non-Python tools (MATLAB, Julia, C/Fortran, h5dump) via a single portable binary file
• Feeding out-of-core training data to machine learning pipelines by memory-mapping slices of an HDF5 dataset on demand
• Consolidating many small per-run data files into one browsable, self-describing hierarchy with attached units/metadata as attributes
• Streaming concurrent writes from a producer process while one or more reader processes tail the file with SWMR

Under The Hood

Architecture h5py is organized in two layers: a low-level Cython layer (h5f, h5d, h5g, h5t, h5s, h5p, h5a, h5r, h5l, h5o, h5z, h5i, h5fd, h5ds, h5pl — one module per corresponding libhdf5 API section, generated from api_functions.txt via api_gen.py) and a high-level Python package under h5py/_hl (files.py, group.py, dataset.py, attrs.py, datatype.py, selections.py, vds.py) that layers Pythonic objects over the low-level identifiers, serialized through a single re-entrant global lock (phil/with_phil in _hl/base.py) since the underlying HDF5 C library is not thread-safe. Group and Dataset subclass a shared HLObject base and implement dict/array-like semantics whose slicing and indexing are translated by selections.py into HDF5 hyperslab and point selections before reaching the low-level h5d calls. This separation means swapping the HDF5 build target (enabling MPI, changing the virtual file driver) doesn’t touch the high-level layer, though the phil locking contract is coarse-grained and ripples through every high-level class that touches it.

Tech Stack Python 3.11+ with a Cython (3.x) extension layer wrapping the C HDF5 library, NumPy for all array interop, and optional mpi4py 4.x integration for parallel builds. The build is driven by a custom in-tree setuptools backend (setup_build.py/setup_configure.py) that detects or links against a system or bundled HDF5 library and regenerates Cython sources from api_functions.txt. Wheels bundling the HDF5 shared library are built and published via GitHub Actions, with additional cross-platform CI on Azure Pipelines and AppVeyor.

Code Quality An extensive pytest suite (30+ files under h5py/tests) covers attributes, datasets, groups, dtypes, filters, SWMR, virtual datasets, ROS3, and MPI/direct-chunk I/O, with custom markers for network/slow/parallel tests and filterwarnings = error so unexpected warnings fail the build. There is no static type-checking layer (the low-level bindings are Cython, not typed Python), but a pylintrc and the zizmor GitHub Actions security linter run via pre-commit. Error handling favors explicit ValueError/TypeError validation close to the HDF5 call sites rather than silent fallback.

API Design The library’s central ergonomic win is presenting HDF5’s entire object model — files, groups, datasets, attributes — as ordinary Python/NumPy objects: File and Group behave like nested dicts, dataset slicing is transparently translated into HDF5 hyperslab selections, and metadata is exposed through a .attrs mapping, so NumPy-familiar users need almost no HDF5-specific vocabulary to get started. Advanced features (SWMR, virtual datasets, direct chunk read/write, custom filters, MPI I/O) are exposed through that same object API rather than a separate low-level-only path, though using them well still requires some HDF5 domain knowledge.