bitsandbytes

bitsandbytes enables accessible large language models via k-bit quantization for PyTorch. Maintained by the bitsandbytes-foundation (originally by Tim Dettmers, now stewarded with Hugging Face), it provides three core capabilities: 8-bit block-wise optimizers that match 32-bit optimizer performance at a fraction of the memory footprint, LLM.int8() 8-bit quantized inference that halves memory requirements without accuracy loss by isolating outlier features into 16-bit matrix multiplication, and QLoRA 4-bit quantization that lets large models be fine-tuned by quantizing frozen weights to 4-bit and training small LoRA adapters on top.

The library ships as drop-in bitsandbytes.nn.Linear8bitLt and bitsandbytes.nn.Linear4bit layers plus an bitsandbytes.optim module of 8-bit optimizer implementations (Adam, AdamW, Lion, LAMB, SGD, and more), and is the quantization backend used by Hugging Face Transformers, PEFT, and Diffusers. It compiles native CUDA/C++/Triton kernels across NVIDIA, AMD, and Intel GPUs plus CPU backends, targeting the exact bottleneck that keeps large models out of reach: GPU memory.

What You Get

• Linear8bitLt and Linear4bit drop-in replacements for torch.nn.Linear that quantize weights transparently during forward passes
• A full bitsandbytes.optim suite of 8-bit optimizers (Adam, AdamW, Lion, LAMB, LARS, RMSprop, SGD, AdEMAMix) that hold optimizer state in 8-bit block-wise format
• LLM.int8() vector-wise 8-bit matrix multiplication with automatic outlier feature detection and 16-bit fallback for those columns
• QLoRA-ready 4-bit quantization (QuantState, NF4/FP4 formats) for fine-tuning frozen quantized base models with trainable LoRA adapters
• A pluggable multi-backend dispatch layer (bitsandbytes/backends/) covering CUDA, CPU, ROCm, Intel XPU, Intel Gaudi (HPU), and Apple MPS
• StableEmbedding, a variance-stabilized embedding layer for more stable low-precision training

Common Use Cases

• Loading a multi-billion parameter LLM for inference on a single consumer GPU via load_in_8bit/load_in_4bit through Hugging Face Transformers
• Fine-tuning large models with QLoRA on limited VRAM by quantizing the base model to 4-bit and training LoRA adapters
• Reducing optimizer state memory during full-parameter training by swapping in 8-bit Adam/AdamW/Lion optimizers
• Running inference or training across heterogeneous hardware (NVIDIA, AMD, Intel GPU/CPU/Gaudi, Apple Silicon) through the shared backend API

Under The Hood

Architecture bitsandbytes is organized as a thin PyTorch-facing API (bitsandbytes/nn/modules.py, bitsandbytes/optim/, bitsandbytes/functional.py) sitting on top of bitsandbytes/_ops.py, which registers custom torch.library ops (e.g. bitsandbytes::int8_scaled_mm, bitsandbytes::int8_mixed_scaled_mm) with fake-tensor implementations for tracing/compile compatibility, and a bitsandbytes/backends/ dispatch layer with dedicated cuda/cpu/hpu/mps/triton/xpu/default implementations selected at runtime based on available hardware. Native compute-heavy kernels live in csrc/ (CUDA .cu/.cuh files for int8 and 4-bit GEMM variants tuned per SM generation, plus C++ CPU ops) and are built via CMake/scikit-build-core rather than pure setuptools. nn.Linear8bitLt/Linear4bit intercept the standard torch.nn.Linear forward path to route through these quantized ops transparently, so swapping precision requires no changes to surrounding model code — the core abstraction that would break if changed is this ops/backend dispatch boundary, since it decouples the Python API from hardware-specific kernel implementations.

Tech Stack The project targets Python 3.10+ and PyTorch 2.4+, with numpy and packaging as its only pure-Python runtime dependencies — all heavy lifting is native code. The build system is scikit-build-core driving CMake to compile CUDA, C++, and some Triton kernels into a shared libbitsandbytes library bundled as package data; this replaces a historical raw setup.py/prebuilt-binary approach. Test/dev extras pull in pytest, transformers, scipy, einops, and lion-pytorch for cross-checking optimizer/quantization correctness against reference implementations, and ruff handles linting across bitsandbytes, tests, and benchmarking.

Code Quality The tests/ directory is extensive, with dedicated suites for functional kernels, autograd correctness, both linear layer types, every optimizer, and CUDA setup detection (test_functional.py and test_optim.py alone exceed 1,700 lines), run via pytest with markers to skip slow/benchmark/deprecated cases in CI. Ruff lint rules are broad (bugbear, pycodestyle, pyflakes, isort, updated-syntax) with narrowly scoped per-file ignores rather than blanket suppression, and the package ships a py.typed marker with mypy overrides configured for a handful of loosely-typed third-party deps, indicating type-checked first-party code. GitHub Actions run lint, PR tests, and nightly tests separately, and the project maintains a CHANGELOG.md and CONTRIBUTING.md alongside per-hardware compile docs (e.g. COMPILE_H100_L40.md).

What Makes It Unique Unlike general quantization toolkits, bitsandbytes’ distinguishing technical choice is treating outlier features as a first-class problem in 8-bit inference (LLM.int8()) rather than quantizing uniformly, which is what preserves accuracy at 8-bit precision for large transformer models — this outlier-isolation technique, published as research from the maintainers, is what QLoRA and much of the Hugging Face quantization ecosystem builds on. Its multi-backend abstraction (extending well beyond CUDA to ROCm, Intel XPU/Gaudi, and Apple MPS) is also unusually broad for a kernel-heavy quantization library, most of which stay CUDA-only.