llama-cpp-2

llama-cpp-2 is a Rust crate that provides safe wrappers around near-direct bindings to llama.cpp, the C/C++ inference engine behind much of the local large-language-model ecosystem. It lets Rust developers load GGUF models and run token-by-token inference on CPU or GPU without writing any FFI code themselves.

The project’s explicit goal is to stay as close to llama.cpp as possible so it can track that very fast-moving upstream, exposing model loading, context and KV-cache management, batching, sampling, grammars, and GGUF metadata parsing through idiomatic-enough Rust types while keeping the surface thin.

What You Get

• Safe Rust types for llama.cpp backends, models, contexts, batches, and samplers
• GGUF model loading with metadata inspection and tokenizer access
• Hardware acceleration behind feature flags (CUDA, Metal, Vulkan, OpenCL, OpenMP)
• Grammar-constrained and JSON-schema sampling support
• The companion llama-cpp-sys-2 crate with the raw FFI bindings kept in lockstep

Common Use Cases

• Embedding local LLM inference directly into a Rust application or service
• Running quantized GGUF models offline on CPU or consumer GPUs
• Building custom sampling or grammar-constrained generation pipelines
• Prototyping LLM tooling that needs to track bleeding-edge llama.cpp features

Under The Hood

Architecture — The workspace splits into llama-cpp-sys-2, which builds llama.cpp and generates raw FFI bindings, and llama-cpp-2, which layers safe Rust abstractions on top. The public crate (src/lib.rs) organizes the API into modules for the backend (llama_backend.rs), models (model.rs), contexts and KV cache (context/, context/kv_cache.rs, context/session.rs), batching (llama_batch.rs), sampling (sampling.rs), grammars (grammar/), GGUF parsing (gguf/), and tokens (token.rs), with a single LlamaCppError enum funneling failures into a crate-wide Result type. Tech Stack — Written in Rust (edition 2021) against llama.cpp’s C API through the sibling -sys crate, using thiserror for error types, tracing for instrumentation, enumflags2 for flag handling, and encoding_rs for text; optional dependencies llguidance and toktrie back grammar-constrained sampling, and a build.rs plus test-build.Dockerfile handle the native compilation. Code Quality — Unit tests live alongside modules (grammar/tests.rs, gguf/tests.rs, model/params.rs, llama_backend.rs and others), errors are modeled explicitly rather than panicking, and types are strongly typed with NonZero and newtype wrappers; the maintainers are candid that the API deliberately stays close to raw bindings rather than chasing full idiomatic polish. API Design — The surface is thin and predictable, mirroring llama.cpp’s own concepts (backend → model → context → batch → sample), which is easy to follow for anyone familiar with llama.cpp but requires more manual wiring than a high-level SDK; documentation is served through docs.rs and a maintained simple example, and the crate intentionally does not follow semver, so version pinning matters.