gguf

gguf is the official Python implementation of GGUF (GGML Universal File), the binary file format llama.cpp uses to store quantized language models, their tensors, and all associated metadata in a single self-describing file. It lives inside the llama.cpp monorepo as the gguf-py package but ships to PyPI as a standalone dependency so conversion tools, inference frontends, and model-hub tooling can read and write GGUF files without linking against the C++ engine.

The package exposes a GGUFReader that memory-maps a file and lazily exposes its key-value metadata and tensor data, and a GGUFWriter that builds a file from scratch — including support for splitting oversized models into multiple shard files. A large constants module encodes the format’s value types, ~30 GGML quantization types (including k-quants and i-quants like iq2_xxs and mxfp4), and a per-architecture tensor-name mapping table covering well over 100 model architectures, so the same reader/writer API works uniformly across LLaMA, Mistral, Gemma, Qwen, and dozens of other model families.

Beyond the core library, gguf bundles five command-line tools (gguf-dump, gguf-convert-endian, gguf-set-metadata, gguf-new-metadata, and a Qt-based gguf-editor-gui) for inspecting and editing existing GGUF files without regenerating them, and it is the library that convert_hf_to_gguf.py uses to turn Hugging Face checkpoints into the .gguf files consumed by llama.cpp, Ollama, LM Studio, and most other local-LLM runtimes.

What You Get

• GGUFReader - memory-maps a .gguf file and exposes its key-value metadata fields and tensors without loading the whole file into memory
• GGUFWriter - builds a GGUF file from tensors and metadata, with support for splitting large models into multiple shard files
• ~30 quantization codecs - bit-exact Python (de)quantization for GGML’s k-quant and i-quant formats (q4_K, q6_K, iq2_xxs, mxfp4, nvfp4, and more)
• Architecture tensor mapping - a canonical name-mapping table translating tensor names across 100+ supported model architectures
• Command-line tools - gguf-dump, gguf-convert-endian, gguf-set-metadata, and gguf-new-metadata for inspecting and editing GGUF files from the shell
• Qt-based GGUF editor GUI - an optional visual tool (pip install gguf[gui]) for browsing and editing a file’s metadata and tensor list

Common Use Cases

• Converting Hugging Face checkpoints to GGUF - convert_hf_to_gguf.py and similar tools use GGUFWriter to produce the .gguf files llama.cpp-based runtimes load
• Building custom model-conversion pipelines - projects outside llama.cpp import gguf directly to emit valid GGUF files for their own quantization or fine-tuning workflows
• Inspecting and debugging GGUF files - gguf-dump and GGUFReader let you audit a file’s metadata, tensor shapes, and quantization types before deploying it
• Patching metadata without re-quantizing - gguf-set-metadata and gguf-new-metadata let you correct a wrong chat template, context length, or tokenizer field on an existing file in place

Under The Hood

Architecture The package splits cleanly into a GGUFReader (gguf_reader.py) that memory-maps the target file via np.memmap and walks the header — magic, version, tensor/KV counts — building ReaderField/ReaderTensor records lazily rather than eagerly materializing tensor data, and a GGUFWriter (gguf_writer.py) that tracks its own write-phase state machine (WriterState: NO_FILE → EMPTY → HEADER → KV_DATA → TI_DATA → WEIGHTS) and can split output across multiple shard files once tensor count or size limits are hit. A sprawling constants.py (~5,800 lines) centralizes the value-type/quantization-type enums and a Keys namespace of standardized metadata key strings, while tensor_mapping.py (~2,900 lines) is a static per-architecture table mapping every supported model family’s native tensor names onto GGUF’s canonical names — this table, together with constants.py, is what has to be extended every time llama.cpp adds support for a new model architecture, making it the package’s central point of ongoing maintenance load.

Tech Stack Pure Python (>=3.10) with a small, deliberately minimal dependency set — numpy for tensor and binary I/O, tqdm, pyyaml, and requests — plus an optional PySide6 extra (gguf[gui]) powering the Qt-based metadata editor. It builds via poetry-core and ships five console-script entry points registered in pyproject.toml. Despite being developed in-tree inside the much larger C/C++/CUDA llama.cpp monorepo, it publishes to PyPI independently through a GitHub Actions workflow triggered by gguf-vX.Y.Z tags.

Code Quality The package is fully typed (from __future__ import annotations, numpy typing, ships py.typed for downstream type-checkers) and covered by unittest-based tests, including an unusually rigorous cross-check in test_quants.py that loads the compiled libggml C library via ctypes and compares every Python (de)quantization routine against the reference C implementation for bit-exact correctness. Parsing code favors explicit ValueErrors with descriptive messages over silent failures, and enforces security-conscious caps on string and array lengths read from untrusted files. A few FIXME comments and commented-out code paths remain around multi-dimensional array handling in the reader, but the core read/write paths are solid.

What Makes It Unique GGUF’s core innovation is packaging everything a runtime needs — quantized weights, architecture hyperparameters, and tokenizer vocabulary — into one memory-mappable file, eliminating the sidecar config.json/tokenizer.json files that formats like safetensors still require. The gguf package is the reference implementation of that format’s exotic quantization schemes (k-quants, i-quants, mxfp4, nvfp4) in pure Python with bit-exact parity to the C decoder, which is what has let GGUF become the de facto interchange format across llama.cpp, Ollama, LM Studio, and most other local-LLM runtimes.