Opik

Opik is the Python client for Comet’s open-source LLM evaluation platform. It gives teams building LLM applications, RAG pipelines, and agentic workflows a single SDK for capturing traces, running automated evaluations, and enforcing guardrails in production. A @track decorator instruments any function to emit structured traces without touching business logic, while a background streaming pipeline batches and uploads telemetry asynchronously so instrumentation never blocks the calling code.

Beyond tracing, the SDK ships a full evaluation engine: heuristic metrics, LLM-as-judge scorers, RAGAS-based metrics, and conversation/trajectory-level evaluators, all runnable against datasets or live experiments through the same client. A guardrails module adds PII detection, prompt-injection detection, and topic checks that can run inline during a traced call. Native integrations cover the major LLM and agent ecosystems — OpenAI, Anthropic, Bedrock, LangChain, LlamaIndex, CrewAI, DSPy, Haystack, and more — so most projects can adopt tracing with a one-line wrapper rather than manual instrumentation.

The SDK also bundles a CLI (opik configure, plus export/import/migration commands for moving trace and dataset data between projects and environments) and works identically against Comet’s hosted cloud or a fully self-hosted Opik deployment.

What You Get

• A @opik.track decorator that instruments any Python function for distributed tracing with zero changes to its logic
• An asynchronous, batching message-streaming pipeline so trace uploads never block application code
• An evaluation engine with heuristic metrics, LLM-as-judge scorers, RAGAS metrics, and conversation/trajectory evaluators
• A guardrails module for inline PII detection, prompt-injection detection, and topic/policy checks
• Prebuilt integrations for OpenAI, Anthropic, Bedrock, LangChain, LlamaIndex, CrewAI, DSPy, Haystack, Groq, Mistral, and more
• A CLI (opik configure, plus export/import/migrate/backup commands) for moving traces, datasets, and prompts between projects

Common Use Cases

• Instrumenting an LLM or RAG application to capture end-to-end traces of prompts, model calls, and intermediate steps
• Running automated regression evaluation of prompts or agent trajectories against a fixed dataset before shipping changes
• Adding inline guardrails (PII, prompt injection, topic restriction) to a production LLM endpoint
• Tracing multi-agent or multi-step agentic workflows built with LangChain, LlamaIndex, or CrewAI without custom instrumentation code
• Exporting traces and datasets from a self-hosted Opik instance for backup or migration to another environment

Under The Hood

Architecture The SDK’s core design cleanly separates instrumentation from delivery: opik/decorator/base_track_decorator.py and opik/opik_context.py capture spans as method calls happen, and instead of sending data inline, they push BaseMessage objects onto a message_queue.MessageQueue consumed by a background Streamer (src/opik/message_processing/streamer.py) that runs attachment preprocessing, batching preprocessing, and file upload through pluggable QueueConsumer workers, with a ReplayManager providing fallback delivery when the queue backs up. This decouples user code from network latency entirely and would need reworking only if the queue/streamer contract itself changed. The evaluation/ and guardrails/ packages sit alongside tracing as independent subsystems that share the same REST client (rest_api/, itself generated code) and context/decorator machinery, and the runner/ package layers a supervised process/TUI mode (bridge handlers, file watcher, PID file, stability guard) on top for long-running local agents.

Tech Stack The package targets Python 3.10+ and builds on httpx for transport, pydantic/pydantic-settings for config and payload validation, litellm for provider-agnostic LLM calls inside evaluation and guardrail judges, tenacity for retries, rapidfuzz for fuzzy matching, sentry_sdk for internal error telemetry, and tree-sitter (with JS/TS grammars) for parsing code embedded in bridge handlers. A generated rest_api/ client talks to the Opik backend, click powers the opik CLI, and watchfiles backs the file-watching runner mode. Packaging is plain setuptools, versioned from a shared repo-root version.txt so the Python SDK stays in lockstep with the rest of the monorepo (backend, TypeScript SDK, frontend).

Code Quality The SDK carries 432 test files across unit, integration, e2e, e2e-smoke, and cross-version-compatibility suites, run via dedicated GitHub Actions workflows (python_sdk_unit_tests.yml, python_sdk_e2e_tests.yml, python_sdk_compatibility_v1_e2e_tests.yml). Type checking is strict: mypy is configured with disallow_untyped_defs, disallow_untyped_calls, and check_untyped_defs all enabled, and ruff enforces lint/format rules via a dedicated .ruff.toml. The codebase follows an explicit internal style guide (documented in the SDK’s own README) covering SOLID principles, module-import conventions, and protected-member discipline, which shows up consistently in the actual module layout.

API Design The public surface favors minimal ceremony: a single @opik.track decorator adds tracing to an existing function, opik.configure() handles both cloud and self-hosted setup in one call, and dynamic-control functions (set_tracing_active, is_tracing_active, reset_tracing_to_config_default) let callers toggle behavior at runtime rather than through redeploys. Framework integrations follow a consistent per-library adapter pattern (opik/integrations/<library>/) so switching from OpenAI to Anthropic or LangChain requires swapping an import, not relearning an API, and the bundled examples/ directory pairs nearly every capability (evaluation, guardrails, distributed tracing, dynamic tracing) with a runnable script.