llama-cpp-python

llama-cpp-python provides Python bindings for the llama.cpp inference library, letting you run quantized large language models locally on CPU or GPU. It exposes both a low-level ctypes interface to the C API and a high-level, Pythonic API for text and chat completions that mirrors the OpenAI client shape.

Beyond the library API, the package ships an OpenAI-compatible web server with support for function calling, vision models, and serving multiple models at once. It integrates cleanly with ecosystems like LangChain and LlamaIndex, and builds llama.cpp from source with configurable hardware-acceleration backends.

What You Get

• High-level Llama class for text and chat completions with an OpenAI-like shape
• Low-level ctypes access to the full llama.cpp C API for fine control
• A drop-in OpenAI-compatible web server with function calling and vision support
• Grammar-constrained generation, tokenizers, caching, and speculative decoding helpers
• Configurable hardware backends (CUDA, Metal, ROCm, etc.) via CMake build options

Common Use Cases

• Running quantized local LLMs on a laptop or server without a cloud API
• Serving a private OpenAI-compatible endpoint for internal apps and tooling
• Powering LangChain or LlamaIndex pipelines with a local inference backend

Under The Hood

Architecture - The package layers a Pythonic API over the native engine: llama_cpp/llama_cpp.py is the generated ctypes binding to the C API, _internals.py/_ctypes_extensions.py wrap resource lifetimes, and llama.py exposes the high-level Llama class with completion, embedding, and chat methods. Chat behavior is pluggable through llama_chat_format.py, grammar-constrained decoding via llama_grammar.py, and an optional FastAPI app under llama_cpp/server/ provides OpenAI-compatible routes. llama.cpp itself is pulled in as a git submodule under vendor/ and compiled at install time.

Tech Stack - Python 3.8+ binding to C/C++ llama.cpp, built with scikit-build-core and CMake (CMakeLists.txt), configurable via CMAKE_ARGS for CUDA/Metal/ROCm/Vulkan backends. The server extra uses FastAPI, Uvicorn, and Pydantic; numpy and typing-extensions support the core API.

Code Quality - A tests/ suite exercises the high-level API and chat formats, CI runs on GitHub Actions, and py.typed ships type hints. The ctypes layer is inherently low-level and verbose but isolated from the ergonomic public surface.

API Design - The high-level API deliberately mirrors OpenAI (create_chat_completion, create_completion), which makes migration and LangChain/LlamaIndex integration nearly frictionless. The main friction is the native build step and backend configuration; once installed, instantiating Llama(model_path=...) and calling completion is straightforward and well documented on Read the Docs.