langchain-litellm

langchain-litellm is the LangChain-maintained integration package that wraps LiteLLM inside LangChain’s standard chat model, embeddings, and document loader interfaces. Rather than writing separate LangChain adapters for every LLM vendor, teams point a single ChatLiteLLM or ChatLiteLLMRouter model at LiteLLM’s unified completion API and get streaming, tool calling, structured output, and token usage accounting that behaves the same across OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Hugging Face, Replicate, and dozens of other providers LiteLLM supports.

Beyond chat, the package exposes LiteLLMEmbeddings and LiteLLMEmbeddingsRouter for provider-agnostic embedding generation with retry handling built in, and a LiteLLMOCRLoader document loader that talks to a LiteLLM proxy’s OCR endpoint (for example, Azure Document Intelligence) to turn files, URLs, or raw bytes into LangChain Document objects. The router-backed classes (ChatLiteLLMRouter, LiteLLMEmbeddingsRouter) delegate to a LiteLLM Router instance so applications can layer load balancing, fallback models, and retry/cooldown policies underneath a single LangChain-facing interface.

Maintained under the langchain-ai GitHub organization with an active release cadence (24 releases, roughly weekly commit activity), it targets teams already standardized on LangChain’s Runnable abstractions who want LiteLLM’s provider breadth without hand-rolling their own adapter layer.

What You Get

• ChatLiteLLM — a LangChain BaseChatModel that calls litellm.completion()/acompletion(), supporting sync/async invoke, streaming, batch, tool/function calling, structured output via with_structured_output, and reasoning-content pass-through for models that emit it
• ChatLiteLLMRouter — the same chat interface backed by a LiteLLM Router, adding load balancing, fallbacks, and cooldowns across multiple deployments or provider keys without changing calling code
• LiteLLMEmbeddings and LiteLLMEmbeddingsRouter — LangChain Embeddings implementations over litellm.embedding(), with built-in retry decorators and separate document/query input-type handling for providers like Cohere and Voyage
• LiteLLMOCRLoader — a LangChain document loader that posts files, URLs, base64, or raw bytes to a LiteLLM proxy’s OCR endpoint and returns single- or per-page Document objects with configurable timeout and retry behavior
• Usage metadata normalization — token counts and provider-specific extras are converted into LangChain’s standard UsageMetadata shape regardless of which underlying provider answered the call

Common Use Cases

• Swapping LLM providers (OpenAI, Anthropic, Azure, Bedrock, etc.) in an existing LangChain app by changing a model string instead of rewriting the integration layer
• Running LangChain agents against self-hosted or enterprise-proxied models through a LiteLLM proxy, keeping API keys and routing logic centralized outside application code
• Adding automatic fallback between model deployments (e.g., primary provider down → secondary provider) via ChatLiteLLMRouter without custom retry logic in the app
• Generating embeddings across multiple providers with one consistent LangChain Embeddings interface for vector store ingestion pipelines
• Extracting text from scanned documents or PDFs into LangChain Document objects using a LiteLLM-proxied OCR provider, for downstream RAG pipelines

Under The Hood

Architecture The package is a thin LangChain-facing adapter layer over LiteLLM: ChatLiteLLM in chat_models/litellm.py subclasses LangChain’s BaseChatModel and centralizes message conversion (_convert_dict_to_message, _convert_delta_to_message_chunk, _convert_message_to_dict), retry decoration, and usage-metadata normalization (_create_usage_metadata); ChatLiteLLMRouter extends ChatLiteLLM and swaps the underlying litellm.completion() call for a pre-built LiteLLM Router instance, reusing the parent’s streaming and conversion logic rather than duplicating it. The embeddings side mirrors this same base-plus-router pattern in embeddings/litellm.py and embeddings/litellm_router.py. The OCR document loader is architecturally independent, implementing LangChain’s BaseLoader directly with its own HTTP timeout/retry handling against a LiteLLM proxy endpoint rather than the LLM completion path. The package has no persistence or app-lifecycle concerns of its own — it exists purely to translate between LangChain’s model interfaces and LiteLLM’s unified provider API.

Tech Stack Built for Python 3.10-3.14 with a hatchling build backend and dependency-managed via uv (checked-in uv.lock). Core dependencies are langchain-core (>=1.4.7), litellm (>=1.83.14, with two explicit version exclusions), httpx, and cryptography. Tooling includes ruff for linting, mypy in strict mode (disallow_untyped_defs), and codespell for spell-checking, all invoked through a Makefile.

Code Quality Each of the five public classes has a dedicated unit test file (test_litellm, test_router, test_embeddings, test_embeddings_router, test_ocr) plus a mirrored integration_tests suite. Notably, test_standard.py and test_standard_router.py run LangChain’s own langchain-tests conformance suite against ChatLiteLLM, validating the integration against LangChain’s shared contract rather than relying solely on ad hoc assertions. Testing infrastructure includes pytest-asyncio, syrupy snapshot testing, blockbuster for catching accidental blocking calls in async code, and responses/requests-mock for HTTP mocking. CI runs a lockfile-consistency check plus a test matrix across Python 3.10-3.13 on every push and PR.

API Design The public surface is deliberately small — five classes exported from the top-level package (ChatLiteLLM, ChatLiteLLMRouter, LiteLLMEmbeddings, LiteLLMEmbeddingsRouter, LiteLLMOCRLoader) — and each mirrors the constructor shape of LangChain’s other provider integrations, so migrating an existing chain from a single-provider chat model typically means changing an import and a model string, not restructuring calling code. The router variants require a pre-built LiteLLM Router object but otherwise preserve the same call surface as their non-router counterparts through inheritance. The OCR loader’s dependency on a separately running LiteLLM proxy (rather than working standalone) is the one place where the abstraction leaks operational complexity into the caller’s setup.