xFormers

xFormers is a toolbox from Meta AI Research that provides independent, customizable Transformer components you can compose without boilerplate. It ships bleeding-edge building blocks — memory-efficient attention, sparse and block-sparse attention, fused softmax, fused layer norm, and fused SwiGLU — that are often faster and more memory-efficient than mainstream PyTorch primitives.

Used widely across vision, NLP, and generative-model research, xFormers bundles its own optimized CUDA kernels while dispatching to other libraries when appropriate, letting researchers iterate quickly on new architectures without sacrificing performance.

What You Get

• Memory-efficient exact multi-head attention that runs up to 10x faster with far lower memory use
• A collection of fused CUDA operators: fused softmax, fused linear layer, fused layer norm, fused dropout+activation+bias, and fused SwiGLU
• Sparse and block-sparse attention primitives for long-sequence and structured-attention models
• Custom CUDA kernels with automatic dispatch to other libraries when they are faster
• Prebuilt pip wheels matched to specific PyTorch and CUDA/ROCm versions

Common Use Cases

• Accelerating attention in vision and language models during training and inference
• Building custom Transformer architectures from independent, reusable components
• Reducing GPU memory footprint to fit larger batches or longer sequences
• Powering diffusion and generative pipelines that rely on efficient attention

Under The Hood

Architecture - xFormers is organized around independent operators exposed through the xformers.ops namespace, with memory_efficient_attention as the flagship entry point. Components are domain-agnostic building blocks that dispatch at runtime to the fastest available backend — its own CUDA kernels, Triton, or upstream PyTorch — based on the tensor shapes and hardware detected. A python -m xformers.info utility reports which kernels were built and are available for the current install.

Tech Stack - The library is written in Python with substantial CUDA and C++ kernel code, and depends on a matching PyTorch build. Distribution relies on prebuilt wheels published against specific CUDA (12.6/12.8/13.0) and ROCm (7.1) versions via the PyTorch download index; source builds use ninja and require a locally installed, version-matched PyTorch.

Code Quality - The project uses black for formatting, tracks coverage via Codecov, and runs CI on CircleCI, indicating a mature engineering setup. Benchmarks are documented in BENCHMARKS.md and the codebase is actively maintained by a large contributor base from Meta AI Research.

API Design - The public API favors drop-in functions like memory_efficient_attention that mirror standard attention signatures, minimizing the boilerplate needed to adopt it. The main friction is installation: matching PyTorch, CUDA, and driver versions can be involved, and building from source is slow, but prebuilt wheels smooth the common path.