agenta

Agenta is a comprehensive open-source LLMOps platform built for engineering and product teams who need to move from ad-hoc prompt experimentation to systematic, production-grade LLM application development. It brings together prompt management, automated evaluation, human annotation, and full-stack observability in a single platform, eliminating the scattered tooling that slows down AI teams.

At its core, Agenta provides an interactive playground where teams can compare prompts side-by-side against test cases, version configurations with branching and environment promotion, and collaborate with subject matter experts without requiring code changes. The platform supports 50+ LLM providers out of the box including OpenAI, Anthropic, and custom self-hosted models.

For evaluation, Agenta offers flexible test set management sourced from production logs, playground experiments, or uploaded CSVs — paired with over 20 pre-built evaluators, LLM-as-a-judge scoring, and a UI and API for both technical and non-technical reviewers. The human annotation module supports structured feedback workflows that close the loop between production observations and iterative improvement.

On the observability side, Agenta ingests OpenTelemetry traces natively and provides pre-built integrations for popular LLM frameworks and providers. Teams get full trace trees, cost and latency breakdowns, and the ability to save any production span as a test case — creating a tight feedback loop between what happens in production and what gets evaluated in development.

What You Get

• Interactive LLM Playground - Compare multiple prompts and model configurations side-by-side against structured test cases, with real-time output rendering for chat, completion, and custom workflow types.
• Prompt Version Control - Version prompts and full configuration schemas with branching, environment promotion (dev/staging/prod), and rollback — giving teams a Git-like workflow for LLM configuration.
• Multi-Model Support - Experiment with 50+ built-in LLM providers or bring your own custom endpoints, with a unified interface for switching models without changing prompt logic.
• Flexible Test Sets - Build test cases from production traces, playground experiments, or CSV uploads, then organize them into named collections for repeatable evaluation runs.
• Automated Evaluators - Run evaluations with 20+ pre-built evaluators, LLM-as-a-judge scoring, regex matching, JSON schema validation, or custom Python evaluators via API.
• Human Annotation Queues - Route production spans or evaluation outputs to structured annotation queues, collect structured feedback from domain experts, and feed it back into evaluation pipelines.
• OpenTelemetry Native Tracing - Ingest traces from any OTLP-compatible source, visualize full trace trees with span-level cost and latency, and drill into individual LLM calls for debugging.
• Custom Workflow Support - Define arbitrary pipeline configurations beyond simple prompt templates — enabling teams to configure multi-step agents, RAG pipelines, and tool-calling workflows through the same versioning system.

Common Use Cases

• Iterating on RAG pipeline prompts - A data engineer versions retrieval prompts and system instructions separately, runs evaluation against a curated test set, and promotes the winning configuration to production through environment staging.
• Cross-functional prompt collaboration - A product team shares a playground workspace with domain experts who refine prompts directly in the UI, while engineers review changes in a structured diff and approve before deployment.
• Root-causing LLM regressions - An engineer notices quality drop in production, finds the failing span in the observability dashboard, saves it as a test case, and reproduces the failure in the playground to identify the prompt change that caused it.
• Automated evaluation in CI - A team triggers evaluation runs via the Agenta API on every pull request, uses LLM-as-a-judge to score outputs, and blocks merges when pass rates drop below threshold.
• Annotation-driven fine-tuning prep - A team routes low-confidence model outputs to an annotation queue, collects expert labels through the structured UI, and exports the curated dataset for fine-tuning or evaluation benchmark creation.
• Multi-model cost optimization - A startup evaluates the same prompt across GPT-4o, Claude Sonnet, and a self-hosted Llama model using identical test cases, then uses cost and quality scores to pick the right model per use case.

Under The Hood

Architecture Agenta follows a domain-driven, layered architecture split across three primary services: an API backend, a web frontend, and an LLM invocation services layer. The API is organized into discrete domain modules — evaluations, tracing, annotations, testsets, environments, webhooks, and more — each with its own FastAPI router, service class, data access object, and Pydantic model layer. This consistent pattern enforces separation of concerns at the domain level rather than the technical layer level. The OSS core and EE extensions are physically separated into oss/ and ee/ directory trees, with runtime feature detection via an is_ee() guard, allowing the codebase to ship as a single artifact with clean feature gating. The frontend is a Next.js application structured around workspace- and project-scoped routing, with Jotai atoms providing reactive state management that avoids the ceremony of Redux while supporting the complex hierarchical state needed by the playground and evaluation modules.

Tech Stack The backend is Python 3.11+ running on FastAPI with Pydantic v2 for validation and SQLAlchemy 2 with asyncpg for async PostgreSQL access. Alembic manages schema migrations. Background task processing uses TaskIQ with Redis as the broker, handling webhook delivery, evaluation queue processing, and async LLM invocations. Authentication is provided by SuperTokens. The frontend is Next.js 14 with TypeScript, Ant Design components, Tailwind CSS for styling, and Jotai for state management. A set of internal workspace packages — agenta-playground, agenta-ui, agenta-entities, agenta-annotation-ui — each built with Turborepo, provide shared primitives across OSS and EE surfaces. The observability layer speaks native OpenTelemetry via OTLP protobuf ingestion, compatible with OpenLLMetry and OpenInference instrumentation libraries.

Code Quality The codebase maintains a well-structured test suite spanning unit tests (co-located per module), integration tests, acceptance tests, and Playwright end-to-end tests for the web frontend. Unit tests follow clear ARRANGE-ACT-ASSERT conventions and cover domain logic like evaluation queue assignment algorithms, query parsing, and environment flag resolution. CI is configured to run unit tests on every pull request across all packages (API, SDK, web, services) with test result reporting. Python code is formatted with Ruff and type-checked through Pydantic’s strict model validation rather than a separate mypy pass. The frontend uses ESLint with TypeScript strict mode. Error handling is consistent across the API, with custom exception hierarchies and decorator-based exception interception that normalizes HTTP status codes throughout.

What Makes It Unique Agenta’s most distinctive architectural choice is treating the playground as a state machine backed by Jotai atoms that model a directed graph of variant nodes, each carrying its full configuration, execution state, and comparison context independently. This enables synchronous multi-variant comparison without any server-side session state. The observability layer is notable for its native OTLP protobuf ingestion endpoint, which allows any OpenTelemetry-compatible framework to push traces directly without a collector sidecar, while the trace processing pipeline propagates metrics and infers trace types within the same write path. The OSS/EE boundary is enforced purely through directory convention and runtime guards rather than build-time code splitting, which keeps the contribution surface clean and allows self-hosters to inspect exactly what features require a commercial license.

Self-Hosting

Agenta uses a split license: code outside the ee/ directory is MIT licensed, while anything under ee/ is covered by the Agenta Enterprise License. For self-hosters, this means the core platform — prompt management, evaluation, observability, and the full API — is freely usable commercially with no restrictions. You can modify and redistribute the OSS core. The EE directory covers advanced access controls, enterprise SSO, usage metering, and certain managed-tier integrations; using those features in production requires an enterprise agreement with Agenta.

Running Agenta yourself means operating a multi-container stack via Docker Compose. The standard self-hosted setup includes the API service, the Next.js frontend, PostgreSQL, Redis, and a reverse proxy via Traefik. This is a real operational commitment: you are responsible for database backups, PostgreSQL tuning for trace volume, Redis high availability, TLS termination, and keeping containers updated as the project ships multiple releases per week. The codebase moves fast — the changelog shows releases approximately every two to three days — so staying on a recent version requires an active maintenance posture.

Compared to Agenta Cloud, self-hosters give up managed upgrades, built-in high availability, SLA-backed uptime, and the support tier that comes with paid plans. The cloud offering also provides a free tier with no credit card required, making it attractive for teams that want to evaluate the platform before committing to self-hosting infrastructure. For teams with strict data residency requirements, self-hosting is the right path; for teams prioritizing iteration speed and low operational overhead, the cloud tier eliminates all infrastructure concerns while preserving access to the full OSS feature set.