LiteLLM

LiteLLM is an open-source Python library that gives you a single, consistent way to call more than 100 large-language-model providers — OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Vertex AI, vLLM, and many more — all in the familiar OpenAI request/response format. Instead of juggling a different SDK, auth pattern, and error type for every model, you import one completion() function and swap providers by changing a model string.

Beyond the core SDK, the same project ships an AI Gateway (proxy server) that adds virtual keys, spend tracking, load balancing, guardrails, and logging for teams that need centralized control. The library itself stays lightweight and importable, making it a drop-in translation layer between your application code and whichever LLM you point it at.

What You Get

• A single completion() / acompletion() interface that works across 100+ LLM providers without provider-specific SDKs
• Drop-in OpenAI-format requests and responses, so existing OpenAI code works by changing only the model string
• Built-in streaming, async, embeddings, image, audio, batch, and rerank endpoints
• Consistent exception types mapped from every provider’s native error format
• Cost tracking and token counting across models via a bundled pricing map
• An optional AI Gateway (proxy server) for virtual keys, spend limits, load balancing, and logging

Common Use Cases

• Building provider-agnostic applications that can switch between OpenAI, Anthropic, Gemini, and Bedrock without code rewrites
• Adding fallback and load balancing across multiple LLMs for reliability
• Centralizing spend tracking and rate limiting for an organization’s LLM usage through the gateway
• Prototyping against many models quickly to compare quality, latency, and cost

Under The Hood

Architecture - LiteLLM is layered around a thin public surface in litellm/main.py that exposes completion, acompletion, embedding, and related entrypoints. Each entrypoint dispatches on a provider-prefixed model string into per-provider adapter modules under litellm/llms/, which translate the OpenAI-format request into the provider’s native wire format and map the response and errors back. Cross-cutting concerns — caching (litellm/caching), cost calculation (litellm/cost_calculator.py), logging integrations (litellm/integrations), and exception normalization (litellm/exceptions.py) — are composed via a client decorator applied at function-definition time. The optional proxy/gateway layer wraps this same core in a FastAPI server with routing, virtual keys, and spend tracking.

Tech Stack - The core SDK depends on httpx and aiohttp for transport, openai for format compatibility, pydantic v2 for typed models, tiktoken and tokenizers for token counting, and jinja2 for prompt templating. The proxy extra pulls in FastAPI, Uvicorn/Granian, Redis (rq), boto3, cryptography, and MCP. A Rust core (litellm-rust) accelerates hot paths, and the project targets Python 3.10–3.14. Reproducibility is pinned via uv.lock.

Code Quality - The repository is heavily tested, with over 2,600 Python test files under tests/, and enforces strict static analysis through ruff (strict config plus budgets), basedpyright, and flake8. CI runs via CircleCI with codecov coverage tracking, and the code uses extensive type hints and a large typing surface in the core modules. The scale of the provider matrix makes some modules large, but error handling is centralized and consistently mapped.

API Design - The developer experience is the project’s standout: getting started requires only pip install litellm, setting a provider API key, and calling completion(model=..., messages=[...]), mirroring the OpenAI SDK so existing code ports with near-zero boilerplate. Switching providers is a one-string change, async and streaming follow predictable naming (acompletion, stream=True), and errors surface as familiar OpenAI-style exceptions. Documentation is thorough, with a dedicated docs site, a cookbook/ of examples, and an examples/ directory.