otari

Otari is a self-hosted, OpenAI- and Anthropic-compatible LLM gateway from Mozilla AI. It sits between applications and 40+ model providers behind a single endpoint, so client code targets one URL and one wire format instead of juggling a different SDK, auth scheme, and error format per provider. Requests to /v1/chat/completions, /v1/responses, and /v1/messages are authenticated, budget-checked, dispatched through the any-llm routing layer, logged, and billed, all before or immediately after the call reaches the underlying provider.

Access control is built around three primitives: users (the identity spend and usage attach to), API keys (the credential clients present, each scoped to a user and optionally restricted to a model allow-list), and budgets (spending caps with optional reset periods, enforced by reserving the estimated cost before a request runs rather than reconciling after the invoice arrives). Provider credentials are supplied once as environment variables or added through the built-in admin dashboard, encrypted at rest with a Fernet key, and never exposed to calling clients, who only ever see their own gw- virtual key.

Beyond routing, Otari runs two opt-in tools itself: a sandboxed Python REPL (otari_code_execution) and a web-search tool, giving any model, including open-weight ones with no native tool support, feature parity with what frontier provider APIs expose natively. A request-level guardrails field lets a caller run a prompt-injection (or other) check before the provider is ever called, either in monitor mode, where the verdict is surfaced on a response header, or block mode, where the request is rejected with a 403 and no provider call is made.

Otari runs standalone with its own SQLite or Postgres database, or in hybrid mode connected to the hosted otari.ai platform, which takes over provider routing, auth, and usage tracking and adds multi-provider fallback. It ships as a Docker image, a Docker Compose stack with profiles for the code-execution, web-search, and guardrails services, and one-click templates for Render and Railway, alongside official Python, TypeScript, Rust, and Go client SDKs and a CLI.

What You Get

• Single OpenAI/Anthropic-compatible endpoint - Exposes /v1/chat/completions, /v1/responses, and /v1/messages, so existing OpenAI or Anthropic SDK clients work by pointing base_url at the gateway.
• Virtual API keys - Clients authenticate with gateway-issued gw- keys scoped to a user and an optional model allow-list; real provider credentials never leave the gateway.
• Pre-spend budget enforcement - Per-user budgets reserve the estimated cost of a request before it runs, rather than reconciling spend after the bill arrives.
• Built-in usage and spend tracking - Every request across every model and app is logged and queryable through /v1/usage, with a dashboard for spend and usage analytics.
• Sandboxed built-in tools - An opt-in otari_code_execution Python REPL and a web-search tool give any model, including open-weight ones, tool parity with frontier provider APIs.
• Request-level guardrails - A guardrails field runs checks like prompt-injection detection before the provider call, in monitor or block mode.
• Standalone or hybrid deployment - Run entirely self-hosted against SQLite/Postgres, or connect to the hosted otari.ai platform for managed routing and multi-provider fallback.

Common Use Cases

• Centralizing multi-provider LLM access - Teams running models across OpenAI, Anthropic, and dozens of other providers put Otari in front of all of them so application code only ever integrates once.
• Metering internal AI tools - Companies issue a scoped gw- key per internal tool or team, capping spend with per-user budgets instead of trusting each team to track provider bills manually.
• Giving open-weight models tool parity - Self-hosted or open-weight models gain code execution and web search via Otari’s built-in tools, matching what closed frontier APIs offer natively.
• Blocking prompt injection before it reaches a provider - Applications handling untrusted user input attach a guardrails: [{"profile": "prompt-injection", "mode": "block"}] field so flagged requests never reach the model.
• Auditing and cost allocation across many apps - Platform teams query /v1/usage to see spend broken out by user, key, and model across every application that shares the gateway.

Under The Hood

Architecture Otari is a layered FastAPI service: main.py wires config, middleware, and routers together at startup, api/routes/ holds thin per-endpoint modules (chat, responses, messages, keys, budgets, users, pricing) that each define a small adapter class (e.g. _ChatAdapter) implementing protocol-specific formatting, while the shared orchestration, auth resolution, budget reservation, provider dispatch, streaming fallback, and tool-loop wiring live once in api/routes/_pipeline.py and are reused across all three generation surfaces. core/ holds config, database, and usage primitives; services/ holds business logic (budgets, aliases, providers, pricing, guardrails, MCP tool loops, file handling); repositories/ and models/ hold the data-access and entity layer. Background asyncio tasks (alias, provider, and price-snapshot refreshers) reload dashboard-driven overrides on a TTL so sibling workers and replicas converge after a write made on another worker. Budget reservation uses a compare-and-swap conditional UPDATE against the database rather than an external lock, so concurrent requests against the same user’s budget cannot double-spend. Swapping out the any-llm dispatch layer or the shared pipeline would ripple through every route, since all three wire protocols funnel through the same attempt-resolution and fallback code.

Tech Stack The server is Python 3.13+ on FastAPI with an async SQLAlchemy ORM (aiosqlite or asyncpg drivers) and Alembic migrations, using any-llm-sdk for the actual multi-provider dispatch, pydantic-settings for configuration, cryptography (Fernet) for encrypting stored provider credentials, prometheus-client for metrics, and OpenTelemetry for tracing/OTLP ingestion. It ships a otari CLI (gateway.cli:main) for serving, migrating, and initializing the database, and deploys via Docker, Docker Compose (with profiles for optional services), and templates for Render and Railway. The admin dashboard is a separate React 19 + HeroUI v3 single-page app built with Vite and TanStack Query, whose compiled bundle is committed into the Python package so the published image and PyPI artifact serve it without a separate build step.

Code Quality The repository has an extensive unit and integration test suite exercising services, routes, CLI behavior, and database migrations, run with pytest plus pytest-asyncio, pytest-xdist, and testcontainers for real Postgres integration coverage; the web dashboard has its own Vitest unit tests and Playwright end-to-end suite. Ruff enforces linting and import ordering, and mypy runs in strict mode across the source, tests, and scripts. Separate CI workflows gate tests, lint, and typechecking independently on every pull request, alongside a Docker build workflow, and a pre-commit config keeps the same checks available locally.

What Makes It Unique What distinguishes Otari from a typical LLM proxy is that it treats tool support as something the gateway itself can provide rather than something every provider must already have: a sandboxed code-execution tool and a web-search tool are implemented once in the gateway and made available to any model behind it, including open-weight models with no native tool-calling story, giving them the same tool-use parity that closed frontier APIs expose. The request-level guardrails field is similarly gateway-owned rather than provider-dependent, checked before the call is dispatched and invisible to the model itself. The standalone/hybrid split, where the identical open-source gateway can run fully self-contained or delegate routing and fallback to a paid hosted platform without any code change or feature gating in the self-hosted path, is a comparatively unusual take on the open-core model.

Self-Hosting

Licensing Model Apache License 2.0, a fully permissive license; every feature in this repository is available in self-hosted deployments with no license keys or paid tiers.

Self-Hosting Restrictions None found. All core functionality, users, keys, budgets, usage tracking, built-in tools, and guardrails, plus the admin dashboard, ships in the open-source repository with no gated code paths, ee/, enterprise/, or pro/ directories, or license-check logic anywhere in the source.

Enterprise Features Not applicable. There is no separate enterprise or paid tier in this repository.

Cloud vs Self-Hosted Otari also powers the hosted otari.ai platform, a separate commercial product. Connecting a self-hosted Otari instance to otari.ai in “hybrid mode” hands off provider routing, auth, and usage tracking to the platform and adds multi-provider fallback, but this is an optional integration with a separate paid service, not a restriction on the open-source gateway itself, which is fully functional running standalone.

License Key Required No.