Candle

Candle is a minimalist machine learning framework for Rust, created by Hugging Face, with a focus on performance and ease of use. Its core crate, candle-core, provides tensors, automatic differentiation, and pluggable CPU, CUDA, and Metal backends behind a clean, PyTorch-like API.

By removing Python from production workloads and keeping binaries small, Candle enables fast, serverless deployments of models such as LLaMA, Whisper, and Falcon while staying lightweight enough to compile to WebAssembly and run entirely in the browser.

What You Get

• A multi-dimensional Tensor type with a PyTorch-like API for creation, reshaping, indexing, and arithmetic
• Reverse-mode automatic differentiation for model training
• Pluggable CPU, CUDA, and Metal backends selectable at runtime via the Device abstraction
• safetensors, GGUF, NPY, and PyTorch checkpoint loading plus quantized tensor support
• A small dependency footprint that compiles to WebAssembly for in-browser inference

Common Use Cases

• Running LLM and speech models like LLaMA, Whisper, and Falcon in Rust services
• Serverless inference where small binaries and cold-start speed matter
• Training neural networks in Rust using candle-nn on top of candle-core
• Embedding ML inference directly in the browser through WebAssembly

Under The Hood

Architecture - candle-core is organized around a Tensor type (tensor.rs) that wraps a Storage plus a Layout describing shape and strides, with compute dispatched through a backend trait (backend.rs) whose implementations live in cpu_backend, cuda_backend, and metal_backend and are selected at runtime by the Device abstraction (device.rs). Reverse-mode autograd is implemented in backprop.rs, and specialized modules handle convolution (conv.rs), quantization (quantized), and checkpoint I/O (safetensors.rs, pickle.rs, npy.rs).

Tech Stack - The crate is pure Rust and leans on the gemm crate for matrix multiplication, half and float8 for reduced-precision types, safetensors and zip for serialization, rayon for CPU parallelism, and optional cudarc, intel-mkl-src, and objc2-metal features for accelerated backends. thiserror drives its error type and the workspace shares versions across the sibling candle crates.

Code Quality - The repository ships an extensive integration test suite under candle-core/tests (tensor, matmul, grad, conv, quantized, serialization, and indexing tests among others) plus doctests embedded in lib.rs, giving strong coverage of the numerical core. Errors are modeled explicitly with thiserror rather than panics, and modules are cleanly separated by concern.

API Design - The public API deliberately mirrors PyTorch and NumPy, so Tensor::randn, reshape, matmul, and Device::new_cuda read naturally to ML practitioners. Documentation is published on docs.rs, the README opens with a runnable matrix-multiply example, and the candle-examples crate provides end-to-end model demos, keeping the boilerplate needed to get started minimal.