Arize Phoenix

Arize Phoenix is an open-source observability and evaluation platform purpose-built for LLM applications, agents, and RAG pipelines. It traces every step of an LLM call chain — prompts, retrievals, tool calls, and completions — using OpenTelemetry-based instrumentation, then surfaces that trace data in a self-hostable UI for debugging, prompt iteration, and dataset curation.

Beyond tracing, Phoenix ships a companion evaluation library (phoenix-evals) for scoring LLM outputs against custom or built-in metrics, and integrates with popular frameworks (LangChain, LlamaIndex, OpenAI, Anthropic, smolagents) via auto-instrumentation. It runs locally, in a notebook, or as a standalone server, and is maintained by Arize AI, whose commercial product builds on the same tracing model for production-scale deployments.

What You Get

• A self-hostable tracing UI (phoenix server) for inspecting LLM/agent/RAG call traces span-by-span
• OpenTelemetry-based auto-instrumentation for LangChain, LlamaIndex, OpenAI, Anthropic, smolagents, and other popular frameworks
• phoenix-evals, a standalone evaluation library for scoring LLM outputs against built-in or custom metrics
• phoenix-otel, lightweight OpenTelemetry exporters for sending trace data to a Phoenix instance
• Dataset and experiment tracking for iterating on prompts and comparing evaluation runs over time
• A GraphQL API (via Strawberry) and Python client (phoenix-client) for programmatic access to traces and datasets

Common Use Cases

• Debugging why an LLM agent produced an unexpected tool call or final answer by inspecting its full execution trace
• Evaluating RAG pipeline quality (retrieval relevance, answer faithfulness) against a curated dataset before shipping
• Comparing prompt or model versions side by side using recorded experiment runs
• Adding LLM observability to an existing LangChain/LlamaIndex app with a few lines of auto-instrumentation

Under The Hood

Architecture - Phoenix is a monorepo combining a Python backend (src/) built on Starlette/uvicorn serving both a GraphQL API (Strawberry) and a REST surface, a React frontend (app/) rendering the trace-exploration UI, and a set of independently-versioned companion packages (packages/phoenix-client, phoenix-evals, phoenix-otel) that can be installed standalone without the full server; trace ingestion follows the OpenTelemetry data model so any OTel-instrumented app can send spans to a running Phoenix instance.

Tech Stack - Python 3.10-3.14 for the backend (Starlette, uvicorn, Strawberry GraphQL, PyArrow, scikit-learn/pandas/numpy for the evals math), a TypeScript/React frontend under app/ and js/, packaged for distribution via PyPI (arize-phoenix plus the split phoenix-* packages) and Docker/Helm/Kustomize manifests for self-hosted deployment.

Code Quality - The tests/ tree has 300+ Python test files covering the API, evals, and instrumentation integrations, CI is driven through GitHub Actions with tox/pytest, and the project publishes structured CONTRIBUTING.md/DEVELOPMENT.md/AGENTS.md guides plus a CHANGELOG.md maintained via release-please automation — commit velocity is very high (300+ commits/month across 100+ tagged releases), typical of a fast-moving, commercially-backed open-source project.

API Design - The auto-instrumentation model (attach an OTel instrumentor for your framework, point it at a Phoenix endpoint) requires minimal integration code and is consistent across supported frameworks; the tradeoff is a broader surface area to learn across the split packages (core server, evals, otel exporters, client) compared to a single-package library, though each is independently documented.