accelerate

Accelerate is Hugging Face’s library for running unmodified PyTorch training loops across any hardware configuration — single CPU, single GPU, multiple GPUs, or TPU pods — without rewriting the training code for each target. Instead of asking developers to learn a new high-level trainer abstraction, it wraps the model, optimizer, and dataloader through a single Accelerator.prepare() call and lets the rest of a standard PyTorch script stay untouched.

The library also ships an accelerate launch CLI and accelerate config wizard that generate a hardware-aware launch configuration once, so the same script can be invoked identically in a Jupyter notebook, on a workstation, or across a Slurm/multi-node cluster. Under the hood it integrates directly with DeepSpeed, FSDP, Megatron-LM, and mixed-precision (fp16/bf16/fp8) execution, giving teams a single code path from local debugging to production-scale distributed training.

What You Get

• Accelerator class - a single object whose prepare() call wraps model, optimizer, and dataloaders for the current hardware without touching training-loop logic.
• accelerate launch / accelerate config CLI - interactive wizard and launcher that generate and reuse a hardware-specific config instead of hand-writing torchrun or TPU launch commands.
• Mixed precision out of the box - fp16, bf16, and fp8 (via TransformerEngine/torchao) execution controlled by config rather than manual autocast wiring.
• DeepSpeed, FSDP, and Megatron-LM integrations - large-model training strategies configured through DeepSpeedPlugin/FullyShardedDataParallelPlugin instead of separate launch scripts.
• notebook_launcher - spins up distributed training directly from a Jupyter/Colab/Kaggle cell, useful for TPU notebooks that can’t shell out to a CLI.
• big_modeling utilities - init_empty_weights and disk/CPU offloading helpers for loading models too large to fit in GPU memory, used for inference on huge checkpoints.

Common Use Cases

• Scaling an existing PyTorch script - a team with a working single-GPU training loop wraps it with Accelerator.prepare() to run unchanged on 8 GPUs or a TPU pod.
• Fine-tuning large language models - researchers use the DeepSpeed/FSDP plugins to fine-tune multi-billion-parameter models that don’t fit on one GPU.
• Reproducible multi-environment training - an ML engineer runs accelerate config once per machine so the same launch command works on a laptop, a shared server, and a Slurm cluster.
• Framework authors building on top of raw PyTorch - libraries like Transformers’ Trainer and fastai use Accelerate internally so their own users get multi-device support for free.
• Low-memory inference on large checkpoints - engineers use init_empty_weights and automatic device/disk offloading to run inference on models that exceed available VRAM.

Under The Hood

Architecture The library is organized around a small number of composable layers rather than a monolithic trainer: state.py holds the process-group/device singleton (AcceleratorState) that every other module reads from, accelerator.py (the ~4,400-line core) exposes the public Accelerator class whose prepare() dispatches model/optimizer/dataloader wrapping to backend-specific code paths (DeepSpeed, FSDP, Megatron, plain DDP), data_loader.py reimplements PyTorch’s DataLoader to handle distributed sharding and prefetching transparently, and big_modeling.py/hooks.py implement device-placement hooks for offload-based inference. This separation means the same user-facing accelerator.prepare(...) call routes through entirely different execution strategies depending on the active AcceleratorState, and the commands/ package (the accelerate launch/config CLI) is a thin layer on top that only generates the config consumed by state.py — a change to the core abstraction (Accelerator.prepare) has to stay backward-compatible across every backend module in utils/ (deepspeed.py, fsdp_utils.py, megatron_lm.py, transformer_engine.py) simultaneously.

Tech Stack The project targets Python 3.10+ and pins a minimal runtime dependency set — numpy, packaging, psutil, pyyaml, torch>=2.0.0, huggingface_hub>=0.21.0, and safetensors — keeping the base install lightweight, with heavy integrations (DeepSpeed, torchao for fp8, bitsandbytes, transformers, wandb/tensorboard/dvclive/trackio trackers) pulled in only via extras_require groups in setup.py. Packaging is standard setuptools with a src/ layout, console-script entry points register the accelerate, accelerate-config, accelerate-launch, and accelerate-merge-weights CLIs, and CI (.github/workflows/) runs a dedicated build-and-test matrix plus nightly, Gaudi3-scheduled, and self-hosted integration-test workflows on top of the standard test suite.

Code Quality The tests/ directory contains 38 test modules covering the accelerator core, each backend integration (DeepSpeed, FSDP, Megatron, TPU), the CLI, and utility modules, using pytest with pytest-xdist/pytest-subtests/pytest-order for parallel and ordered execution as declared in the test_prod/test_dev extras. Linting and formatting run through ruff (configured in pyproject.toml with an extended PEP8/import-sorting/copyright ruleset and a banned-API check that forbids raw os.getenv calls in favor of os.environ), enforced via .pre-commit-config.yaml and a dedicated quality.yml CI workflow, and the codebase is fully type-hinted in its public API surface.

What Makes It Unique Accelerate’s distinguishing choice is refusing to introduce a new training-loop abstraction: instead of a Trainer.fit()-style API that owns the loop, it wraps only the objects (model, optimizer, dataloader) a developer already has and returns control immediately, so a five-line diff turns a single-GPU script into one that runs unchanged across CPU/multi-GPU/TPU/DeepSpeed/FSDP. That design is also why other frameworks (Transformers’ Trainer, fastai, Catalyst) build on top of it rather than around it — it functions as a distributed-execution substrate other libraries compose with, not a competing high-level framework.