luqum

luqum (LUcene QUery Manipulator) is a Python library that parses queries written in the Lucene Query DSL — the syntax used by Solr, Elasticsearch’s query_string, and countless internal search boxes — into a proper abstract syntax tree instead of leaving you to regex-hack raw query strings. Once parsed, the tree is built from typed node classes (SearchField, AndOperation, Range, Fuzzy, Proximity, Boost, and more) so you can walk it, validate it, or rewrite it programmatically before it ever reaches a search backend.

The library ships a visitor/transformer framework modeled on Python’s own ast module, plus a dedicated ElasticsearchQueryBuilder that turns a parsed tree directly into an Elasticsearch JSON DSL query — including support for nested and object fields, per-field match types, and named sub-queries for inner-hit tracking. A companion LuceneCheck visitor validates a query’s structure (field names, nesting rules) before you ever send it downstream.

A detail that matters in production: luqum preserves whitespace and original formatting (head/tail metadata on every node), so a parsed-then-rewritten query doesn’t silently reformat itself. This makes it practical for building query builders, search-syntax linters, or query rewriters that need to modify only part of a user’s original input.

The project is maintained by Jurismarches and has been in production use since 2016, with a CI matrix that exercises real Elasticsearch instances across versions 6 through 8.

What You Get

• Lucene query parser - a PLY-based lexer/parser that turns Lucene DSL query strings into a typed Python AST (SearchField, AndOperation, OrOperation, Range, Fuzzy, Proximity, Boost, Regex, and more).
• Visitor/transformer framework - TreeVisitor and TreeTransformer base classes, modeled on Python’s own ast module, for walking or rewriting the parsed tree without touching internals of every node type.
• ElasticsearchQueryBuilder - a visitor that converts a parsed tree directly into Elasticsearch JSON DSL, with support for nested/object field mappings, per-field match types, and named sub-queries.
• Query validation - LuceneCheck, a visitor that checks a query’s structural consistency (field names, illegal nesting) before it’s sent to a search backend.
• Format-preserving round-trip - every node carries head/tail whitespace metadata, so parsing and re-serializing a query doesn’t reformat parts you didn’t touch.
• Nested query extraction helper - utilities to pull out and re-nest sub-queries so Elasticsearch reports inner-hit matches correctly under bool/nested queries.

Common Use Cases

• Letting end users type Lucene-style search syntax into a search box, then safely translating it into an Elasticsearch query instead of trusting raw query_string input.
• Building a query linter or validator that rejects malformed or disallowed field references before a search request is issued.
• Rewriting or restricting user queries programmatically — e.g. injecting tenant/permission filters into an existing query tree without altering the rest of the syntax.
• Supporting nested-object search mappings in Elasticsearch, where field names need per-field match-type and analyzer configuration derived from a schema.
• Migrating a legacy Lucene/Solr-syntax search feature onto Elasticsearch without forcing users to relearn a new query language.

Under The Hood

Architecture luqum is a small layered pipeline: a PLY-based lexer/parser (parser.py) turns Lucene DSL text into a generic AST built from Item subclasses declared in tree.py, where each class declares its own _children_attrs/_equality_attrs so traversal and equality are driven by class metadata rather than hand-written per-node logic. visitor.py supplies TreeVisitor/TreeTransformer base classes using camel_to_lower-based method dispatch (mirroring Python’s own ast.NodeVisitor), which check.py’s LuceneCheck and the elasticsearch/ subpackage both build on. The elasticsearch/ subpackage is a second, ES-specific AST (AbstractEItem subclasses exposing a .json property) that ElasticsearchQueryBuilder (a TreeVisitor) produces by walking the generic tree — cleanly separating “parse Lucene syntax” from “emit an ES query” so either side can evolve independently. head_tail.py/auto_head_tail.py thread whitespace-preservation through the pipeline as a cross-cutting concern.

Tech Stack Pure Python 3.10+ with a single runtime dependency, PLY (Python Lex-Yacc) >=3.11, used to generate the lexer and parser tables (parsetab.py is checked into the repo). No web framework, database, or deployment target — it’s a parsing/transformation library. Test-only integrations pull in elasticsearch-dsl to exercise the query builder against real Elasticsearch instances (6.x through 8.x) in CI. Packaged with setuptools for PyPI; documentation is built with Sphinx and hosted on Read the Docs.

Code Quality Extensive tests under tests/ cover the lexer, parser, tree, visitor, check, and Elasticsearch-conversion layers individually, including doctest-based examples (--doctest-modules, --doctest-glob="test_*.rst") that double as living documentation. CI runs a matrix across four Python versions and three Elasticsearch/elasticsearch-dsl version pairs, with branch coverage tracked via pytest-cov and coveralls. Errors are raised as explicit, purpose-built exception types (IllegalCharacterError, ParseSyntaxError, NestedSearchFieldException, ObjectSearchFieldException) rather than generic ones, and a dedicated quality_checks CI job runs linting. Docstrings are consistently present on public classes and methods; there are no static type hints or a type checker configured.

What Makes It Unique Most Lucene-to-Elasticsearch conversion happens ad hoc with string manipulation or regex. luqum instead gives you a real, typed AST with a visitor/transformer API deliberately shaped like Python’s own ast module, so anyone comfortable with ast.NodeVisitor can extend it immediately. Preserving original whitespace/formatting on every node — rather than reconstructing a canonical string — is the detail that makes it viable for query rewriting rather than just query translation, and the nested-query extraction helper solves a specific, easy-to-get-wrong Elasticsearch pain point (inner-hit naming under bool/nested queries) that most hand-rolled query builders don’t address at all.