langchain-nvidia-ai-endpoints

langchain-nvidia-ai-endpoints connects LangChain to NVIDIA’s AI Foundation Models, including the Nemotron family, through the hosted NVIDIA API Catalog or through self-hosted NVIDIA NIM microservices. It ships LangChain-native classes for chat (ChatNVIDIA), completions (NVIDIA), embeddings (NVIDIAEmbeddings), reranking (NVIDIARerank, including vision-language rerank), and RAG document retrieval (NVIDIARAGRetriever), so teams can swap between NVIDIA-hosted inference and on-premises NIM containers without changing application code.

Beyond basic chat/embed/rerank coverage, the package adds NVIDIA-specific plumbing that a generic OpenAI-compatible client would miss: automatic polling for NVCF’s 202/Retry-After async inference flow, multimodal image input handling for VLM models, a ChatNVIDIADynamo variant that injects Dynamo KV-cache routing hints for smarter inference scheduling, and a universal inference_priority decorator/context manager that any Pydantic-based BaseChatModel can pick up to deprioritize background LLM calls relative to interactive ones.

What You Get

• ChatNVIDIA — LangChain chat model wrapping NVIDIA-hosted and self-hosted chat/VLM/reasoning models, with tool calling, structured output, and streaming
• NVIDIAEmbeddings and NVIDIARerank — embedding and reranking clients, including a vision-language rerank path that scores documents carrying an image alongside text
• NVIDIARAGRetriever — a retriever that queries a running NVIDIA RAG Blueprint server’s /v1/search endpoint
• ChatNVIDIADynamo — a drop-in ChatNVIDIA variant that adds NVIDIA Dynamo KV-cache routing hints for Dynamo-enabled deployments
• inference_priority — a context-manager/decorator that injects a priority value into any compatible BaseChatModel call without per-model integration work
• Built-in handling of NVCF’s asynchronous 202 + polling response pattern, so long-running inference calls resolve transparently

Common Use Cases

• Building LangChain chat, RAG, or agent applications against NVIDIA’s hosted Nemotron and other foundation models
• Migrating a LangChain app from the hosted NVIDIA API Catalog to an on-premises NVIDIA NIM deployment for data residency or IP control
• Adding image-aware reranking to a RAG pipeline where some retrieved documents carry images rather than pure text
• Prioritizing interactive user-facing LLM calls over background/batch LLM work sharing the same Dynamo-routed inference cluster
• Querying an NVIDIA RAG Blueprint deployment’s search endpoint directly as a LangChain retriever

Under The Hood

Architecture The package is organized as a flat module built around one shared low-level client. _common.py defines _NVIDIABaseClient, a Pydantic model that owns HTTP request construction, auth-header injection, base-URL resolution, and both sync and async request execution, including the wait loop for NVCF’s asynchronous 202-plus-polling response pattern. chat_models.py’s ChatNVIDIA subclasses this alongside LangChain’s BaseChatModel, layering on payload construction, streaming and generate paths, multimodal image preprocessing for vision-language models, tool-calling, and structured output. embeddings.py, llm.py, reranking.py, and retrievers.py each thinly wrap the same shared client for their respective LangChain base classes, and _statics.py holds the model-capability table that resolves a model name to its profile. decorators.py sits apart as a cross-cutting concern: it patches BaseChatModel’s invoke/stream methods at import time via a context variable, so every model class built on the shared client inherits its network and auth behavior in one place, for better or worse.

Tech Stack A Python 3.10+ package built with Poetry inside a multi-package monorepo alongside sibling NVIDIA integrations. Runtime dependencies are limited to langchain-core, aiohttp for async HTTP, and requests for sync HTTP, with Pydantic v2 models used throughout for validation. Development tooling is comprehensive: pytest with pytest-asyncio, pytest-mock, pytest-watcher, snapshot testing via syrupy, HTTP mocking via requests-mock, time mocking via freezegun, and synthetic data via faker. Static analysis runs ruff for linting and mypy with untyped-def checking enforced, plus codespell for prose. CI is a set of reusable GitHub Actions workflows shared across the monorepo’s packages, covering test, lint, codespell, scheduled tests, and release.

Code Quality The unit and integration test suites are extensive, covering authentication precedence, the async 202-polling protocol, tool binding, model registration, and multimodal message handling, not just happy-path calls. Tests run with strict marker and config enforcement, and type hints are used consistently with mypy’s untyped-def check enforced in CI alongside ruff linting. Error handling in the shared client distinguishes authentication failures, polling responses, and general request errors explicitly rather than swallowing exceptions broadly.

What Makes It Unique Several pieces go beyond generic feature parity with other provider integrations: the inference_priority decorator patches LangChain’s BaseChatModel globally through a context variable, so any Pydantic-based chat model can honor call-scoped priority without per-model wiring; ChatNVIDIADynamo layers NVIDIA Dynamo’s KV-cache routing hints directly into requests for smarter multi-tenant inference scheduling; and NVIDIARerank’s vision-language rerank path scores image-bearing documents directly rather than requiring callers to caption images first.