OpenEvals

OpenEvals is an open-source library from LangChain that gives you a starting point for writing evals for LLM applications, much like tests are for traditional software. It ships a collection of ready-made evaluators, including configurable LLM-as-judge scorers driven by prompt templates and deterministic heuristics such as exact match, JSON matching, and embedding similarity.

Evaluators are simple callables: you create one with a factory like create_llm_as_judge, pass in inputs and outputs, and get back a structured score with an explanation. OpenEvals integrates with LangSmith for tracking and pytest for running evals in CI, and it is published for both Python and TypeScript from a shared monorepo.

What You Get

• LLM-as-judge evaluators created from prompt templates via create_llm_as_judge
• A library of ready-to-use prompts (conciseness, correctness, hallucination, and more)
• Deterministic evaluators for exact match, JSON matching, and embedding similarity
• Trajectory and simulator utilities for evaluating multi-step agent runs
• Integration hooks for LangSmith experiment tracking and pytest-based eval suites

Common Use Cases

• Scoring chatbot or RAG outputs for correctness and conciseness with an LLM judge
• Adding regression evals to CI that fail a build when output quality drops
• Checking that structured (JSON) outputs match an expected schema or values
• Measuring semantic similarity between generated and reference answers

Under The Hood

Architecture — The openevals package is organized by evaluator family: llm.py exposes the create_llm_as_judge factory that binds a prompt template and a model into a callable scorer; exact.py, json/, string/, and code/ provide deterministic and structural evaluators; trajectory/ and simulators/ handle multi-turn agent evaluation. Each evaluator returns a normalized result object (key, score, comment) defined in types.py, so consumers get a consistent shape regardless of the underlying scoring strategy. Prompt templates live under prompts/ as reusable f-strings.

Tech Stack — Python 3.10+, built on langchain, langchain-openai, and langsmith, with rich for output formatting. Packaged with setuptools via pyproject.toml, type-checked with mypy and pyright (ships py.typed), linted with ruff, and tested with pytest and pytest-asyncio. It is part of a monorepo that also publishes a parallel TypeScript implementation.

Code Quality — The repository has a broad test suite (test_conciseness.py, test_correctness.py, test_hallucination.py, test_json_async.py, and more) exercising each evaluator, plus async variants. Type hints are shipped and enforced by two type checkers, and the dev toolchain includes ruff and multiple LLM provider integrations for testing. Development is active with frequent releases.

API Design — The public API is deliberately minimal and composable: an evaluator is just a function you call with inputs and outputs, and factories like create_llm_as_judge let you swap prompts and models without changing call sites. Bundled prompt constants keep the common cases one import away, so getting a first eval running takes only a few lines.