fenic

fenic is a Python DataFrame query engine for semantic data processing. It extends familiar PySpark/SQL-style operations (select, filter, join, group_by, agg) with semantic operators — extract, classify, summarize, embed, and semantic join — that call language models as a first-class part of the query plan rather than as ad hoc side calls.

You configure models once on a Session, build a lazy pipeline, and fenic compiles and executes it on a query engine purpose-built for inference: automatic batching, rate limiting, retries, token/cost accounting, and response caching. Because the pipeline is expressed as typed operators, it is inspectable (row-level lineage, explain plans, per-query metrics), rerunnable, and promotable into a named table, view, or MCP tool that an agent can call directly.

What You Get

• A PySpark/SQL-style DataFrame API (select, filter, join, group_by, agg) extended with semantic operators (semantic.extract, semantic.classify, semantic.reduce, semantic.join, embeddings)
• A Session/SessionConfig model for configuring one or more language model providers (OpenAI, Anthropic, Google, Cohere, OpenRouter) with rate and token limits
• A local execution backend built on Polars and DuckDB, plus Rust-accelerated operators (chunking, jinja templating, JSON/markdown parsing, regex, transcript handling) via a native extension
• Row-level lineage, explain plans, and per-query token/cost metrics for inspecting and debugging semantic pipelines
• Built-in MCP server generation (create_mcp_server, SystemToolConfig) to expose fenic tables and catalog tools directly to agents
• A fenic check static linter and a fenic skill install command that teaches coding agents (Claude Code, Cursor, Codex) fenic’s API conventions

Common Use Cases

• Turning free-text support tickets, transcripts, or logs into typed, queryable rows with semantic.extract against a Pydantic schema
• Triaging and root-causing agent eval failures by classifying traces and summarizing recurring failure patterns per category
• Matching records on meaning rather than exact keys — e.g. candidate resumes against job descriptions — using semantic.join
• Publishing a processed table as an MCP tool so an agent can query governed, typed results instead of raw unstructured data

Under The Hood

Architecture fenic compiles a lazy DataFrame plan (src/fenic/api/dataframe/dataframe.py, ~2,000 lines) into a logical plan (src/fenic/core/_logical_plan) that is transpiled and executed by a local backend (src/fenic/_backends/local) built on Polars and DuckDB, with a separate _backends/cloud path for hosted execution. Semantic operators (_backends/local/semantic_operators) route through an _inference layer that handles per-provider batching, rate limiting, retries, and caching before calling out to OpenAI/Anthropic/Google/Cohere/OpenRouter. Performance-critical primitives — text chunking, Jinja templating, JSON/markdown parsing, regex, and transcript handling — are implemented as a native Rust crate (rust/src) exposed to Polars via pyo3/pyo3-polars plugins, so hot-path text operations run outside the Python interpreter while the query surface stays a normal DataFrame API.

Tech Stack The Python side targets 3.10–3.12 and depends on polars (1.34–1.36), duckdb (1.1–1.5), pyarrow, pandas, numpy, sqlglot, protobuf, tiktoken, and cloudpickle, managed via uv (uv.lock) with maturin building the Rust extension (rust/Cargo.toml: pyo3, polars-arrow, jaq-core/jaq-json for JSON, minijinja, rapidfuzz, tiktoken-rs). A justfile and mise.toml standardize local dev/build commands across the mixed Python/Rust toolchain.

Code Quality The repo carries 135 Python test files under tests/ covering package-size matrices, optional extras, the fenic check linter, and asset validation, plus a rust/src/fuzz fuzzing target for the native parsing code — an unusually thorough testing posture for a library at this stage. A dedicated FenicError exception hierarchy (src/fenic/core/error.py) distinguishes configuration, session, and cloud-session errors rather than surfacing raw exceptions, and ruff.toml plus CI-facing fenic check enforce static consistency on both the library’s own code and user-written pipelines.

API Design The public surface mirrors PySpark/SQL naming (select, filter, join, group_by, agg) so existing DataFrame experience largely transfers, while semantic operators (semantic.extract, semantic.classify, semantic.join, semantic.reduce) are typed against Pydantic schemas so LLM output is validated at plan time rather than parsed ad hoc. The project explicitly acknowledges its own newness by shipping a fenic-mechanics skill (fenic skill install) and a static linter (fenic check) specifically to keep AI coding agents from guessing at its namespace conventions, which is an unusually self-aware piece of developer-experience tooling for a library this young.