LLM Gateway

LLM Gateway is an open-source API gateway that gives developers and enterprises a single OpenAI-compatible endpoint for every major language model provider — OpenAI, Anthropic, Google Vertex AI, AWS Bedrock, Azure, DeepSeek, Mistral, and dozens more. Instead of maintaining separate integrations and API key sets for each vendor, you point your existing SDK at LLM Gateway and let it handle provider selection, failover, and cost tracking centrally.

At its core, the gateway is a Hono-based proxy that authenticates requests against your stored API keys, resolves which provider and model to use, translates the OpenAI-compatible payload to the target provider’s format, streams the response back, and asynchronously logs cost and performance data. A background worker persists logs to PostgreSQL, keeping the hot path latency minimal. Redis caches frequently-read configuration (routing rules, key health state, organization settings) so most requests never hit the database.

The routing layer goes beyond simple round-robin. The gateway tracks a sliding-window health score per API key, permanently blacklists keys that return authentication errors, and uses configurable weights — uptime, latency, cost — to pick the best available provider for each request. Enterprise teams can define per-project routing configurations with custom weights and thresholds, compliance policies that restrict routing to specific geographic regions, and guardrails that evaluate every request against a rule engine before it ever reaches an LLM provider.

The platform ships with a Next.js dashboard for usage analytics, a live chat playground supporting text, image generation, and reasoning models, a dedicated admin interface, and a documentation site. It can be self-hosted as a single unified Docker container or as separate services via Docker Compose, and also supports Helm chart deployment to Kubernetes for teams that need production-grade orchestration.

What You Get

• Unified OpenAI-compatible API - Drop-in replacement for the OpenAI SDK; change only the base URL to route requests through LLM Gateway to any of 25+ supported providers without modifying application code.
• Uptime-aware provider routing - The gateway tracks a sliding-window health score for every API key, temporarily skips keys with consecutive errors, permanently blacklists 401/403 failures, and ranks candidates by a configurable blend of uptime, latency, and cost.
• Real-time cost and token analytics - Per-request cost calculations using Decimal.js for precision, broken down by provider, model, input tokens, output tokens, and reasoning tokens, with per-project usage dashboards and exportable logs.
• Multi-key load balancing - Provide comma-separated values for any provider’s API key environment variable and the gateway automatically distributes requests across keys, maintaining independent health tracking per key.
• Guardrails rule engine - Six built-in safety rules (prompt injection, jailbreak detection, PII, secrets, file type restrictions, document leakage prevention) plus custom blocked terms, regex patterns, and topic restrictions — each configurable to Block, Redact, or Warn.
• Compliance geo-routing - Enterprise routing configurations can restrict which provider regions are eligible per project, enforcing data-residency policies so requests never leave approved geographic boundaries.
• MCP protocol support - The gateway implements the Model Context Protocol alongside the OpenAI chat API, enabling AI coding agents and MCP-aware tools to discover and call tools through the same endpoint.
• Chat playground - A built-in Next.js playground supports text generation, image generation, reasoning models, group chat for side-by-side comparison, and GitHub MCP tool integration, accessible from the cloud or self-hosted dashboard.
• Enterprise team and billing management - Organization-level team roles, audit logs, referral system, Stripe-based credit purchasing with PDF invoice generation, and data retention controls configurable per plan.
• Kubernetes and Docker deployment - Self-host via a single unified Docker image (all services in one container), Docker Compose split deployment, or a Helm chart published to GitHub Container Registry as an OCI artifact.

Common Use Cases

• Multi-provider AI application - A product team uses LLM Gateway to route customer-facing chatbot requests to the cheapest available model while automatically failing over to a backup provider when the primary exceeds error thresholds, all without code changes in their application.
• Cost optimization across providers - An AI-powered SaaS company tracks per-model token costs in the LLM Gateway dashboard, identifies that a specific Anthropic model costs 40% more per output token than a Google Gemini equivalent for their use case, and reconfigures routing weights to prefer the cheaper provider.
• Enterprise compliance enforcement - A healthcare company self-hosts LLM Gateway and configures compliance geo-routing to ensure that patient-related prompts only route to EU-region providers, with guardrails that redact PII before any content reaches an external API.
• AI coding agent infrastructure - A development team routes Claude Code and Cursor requests through LLM Gateway to get a 20% discount on Google models and centralized spend visibility across all developer seats, using the MCP endpoint so agents can also call internal tools through the same gateway.
• Migration from OpenAI to alternatives - A startup running on the OpenAI Python SDK switches to LLM Gateway in an afternoon by changing one base URL, then uses the provider comparison dashboard to benchmark Claude Opus and Gemini Pro against GPT-4o before committing to a new default provider.

Under The Hood

Architecture

LLM Gateway follows a modular monorepo architecture with clear functional decomposition across purpose-built applications and shared workspace packages. The Hono-based gateway handles request authentication, provider protocol translation, and streaming response proxying; a separate Hono API manages users, organizations, billing, and configuration; Next.js apps cover the UI dashboard, admin interface, playground, and documentation site; a background worker processes async log persistence, webhook delivery, and data retention cleanup. Shared packages — db, models, shared, cache, logger, instrumentation — define explicit dependency boundaries enforced by the Turbo build graph. Data flows from client through the gateway, which resolves routing metadata from Redis-cached configuration, dispatches to the target provider, and enqueues log entries for asynchronous persistence, keeping the hot path decoupled from database write latency. The enterprise directory mirrors this structure under a separate commercial license, isolating paid capabilities from the AGPLv3 core without polluting the open-source path.

Tech Stack

The entire platform is TypeScript, running on Bun in Docker for reduced cold-start overhead. The gateway and API use Hono for its lightweight, edge-friendly characteristics; the UI, admin, playground, and docs use Next.js 15/16 with App Router in standalone Docker output. Drizzle ORM with PostgreSQL handles persistent storage, with typed query helpers and Zod-validated schemas generated from the centralized packages/db workspace. Redis via ioredis powers SWR-style caching for hot configuration queries, preferred-provider state, and rate-limiting counters. Turbo orchestrates incremental monorepo builds with remote caching. Vitest covers both unit and E2E test suites, with E2E tests running against a real PostgreSQL instance. OpenTelemetry provides distributed tracing across gateway and API with configurable sampling. Stripe handles billing; Brevo handles transactional email; PostHog provides product analytics.

Code Quality

The codebase has extensive test coverage with over 126 unit and integration spec files plus 23 end-to-end test files covering streaming responses, tool calls, prompt caching, graceful shutdown sequences, and provider fallback scenarios. Error handling is explicit and typed throughout — the gateway classifies every terminal state into a UnifiedFinishReason enum and surfaces it consistently in logs and API responses. Provider authentication errors permanently blacklist keys while transient failures trigger configurable retry logic. API boundaries are validated with Zod schemas via @hono/zod-openapi, ensuring runtime and compile-time safety align. Naming conventions are consistent and domain-oriented, and the SWR cache layer wraps frequently-read DB queries to keep concurrent request latency low without sacrificing consistency.

What Makes It Unique

LLM Gateway’s most technically distinctive contribution is its configurable, uptime-aware routing system. Unlike simple key-substitution proxies, it maintains per-key sliding-window health tracking, aggregates provider latency and error-rate history using a weighted scoring model, and exposes per-project routing configuration overrides in the enterprise tier — letting teams tune the balance between cost, speed, and reliability for each workload independently. The compliance engine enforces data-residency policies at the routing layer, blocking requests from being forwarded to non-approved geographic regions before they leave the host. The guardrails module evaluates every inbound request against a configurable rule engine with Block, Redact, or Warn semantics, making content safety an infrastructure concern rather than an application one. Multi-key load balancing with independent health tracking is configured via environment variables, requiring no database schema changes to add capacity. MCP protocol support alongside the OpenAI-compatible API makes LLM Gateway usable as both a model router and a tool-call proxy for AI coding agents.

Self-Hosting

LLM Gateway is dual-licensed. The core gateway, API, UI, and worker applications are available under AGPLv3, which permits free use and modification but requires that any modifications distributed over a network (including SaaS deployments) be released under the same license. Commercial teams running a modified version as a private internal service or SaaS product should evaluate whether AGPLv3’s network-use copyleft applies to their situation. Enterprise features — guardrails, audit logs, advanced billing, unlimited data retention, compliance geo-routing, team management, and custom provider key configurations — live in the ee/ directory and are licensed under a proprietary Enterprise License that requires a valid subscription from LLMGateway, Inc.

Operationally, self-hosting LLM Gateway requires running PostgreSQL 17, Redis 8, and up to six application containers (gateway, API, UI, admin, playground, docs), or a unified single-container image that bundles all services. The unified image is the simplest entry point and works well for small teams; larger deployments benefit from the split Docker Compose configuration or the published Helm chart for Kubernetes. You are responsible for database backups, container updates, TLS termination, and scaling the gateway horizontally when request volume grows. The gateway exposes several ports and requires at minimum three environment variables (auth secret, API key hash secret, and provider API keys); additional configuration variables cover Stripe, Brevo, PostHog, and SMTP for the full feature set.

Compared to the managed cloud version at llmgateway.io, self-hosting means you handle your own uptime, incident response, and version upgrades. The cloud version receives automatic updates, managed infrastructure, and built-in support channels. The enterprise plan adds priority support SLAs, which are not available to self-hosted deployments without a separate enterprise agreement. The managed version also provides Routeway, LLMGateway’s own provider network with discounted pricing on select models, which is not accessible to self-hosted instances that route directly to upstream providers.