@posthog/hogql-parser

@posthog/hogql-parser compiles PostHog’s HogQL (a ClickHouse-flavored SQL dialect) and Hog (PostHog’s own scripting language) grammar to WebAssembly, exposing parseExpr, parseSelect, parseOrderExpr, parseProgram, parseFullTemplateString, and parseStringLiteralText functions that each return a JSON-serialized AST rather than a live object graph. It shares its ANTLR4 grammar and C++ visitor core with the PyPI package hogql_parser (a native CPython extension) and an in-progress Rust port, so all three interfaces produce identical parse trees regardless of runtime.

Inside PostHog’s own frontend, the package is loaded lazily and run inside a Web Worker rather than on the main thread, because a full parseSelect call costs roughly 36ms per 1000 characters of query text — enough to visibly stall a SQL editor on every keystroke. Parse failures never throw; they resolve to a structured {error, type, message} JSON object, and three separate WASM builds (Node ESM, browser-only ESM, and CJS) are produced from the same C++ source so the grammar works identically whether it’s pulled in by a bundler, a browser worker, or Jest.

What You Get

• Six parse entry points - parseExpr, parseSelect, parseOrderExpr, parseProgram, parseFullTemplateString, and parseStringLiteralText cover expressions, full SELECT statements, ORDER BY clauses, Hog programs, template strings, and raw string-literal unquoting.
• JSON-in, JSON-out API - every parse call returns a JSON string rather than a live JS object graph, keeping the WASM/JS boundary cheap and letting the AST cross a Web Worker postMessage without extra serialization work.
• Non-throwing error handling - syntax and internal parser errors come back as a {error: true, type, message, start, end} JSON object instead of a thrown exception or rejected promise.
• Three build targets from one core - Node ESM, a browser-only ESM stripped of Node-specific import paths, and a CommonJS build for Jest, all compiled from the same parser_wasm.cpp via CMake + Emscripten.
• Full TypeScript definitions - index.d.ts hand-documents every method with JSDoc and runnable examples, including the Position/ASTNode/ParseError shapes callers get back.

Common Use Cases

• Powering a SQL editor’s autocomplete - PostHog’s Monaco-based HogQL editor calls parseSelect on every debounce to build table/column completions and find the innermost SELECT under the cursor.
• Off-main-thread query parsing - because a full parseSelect can take seconds on a long query, apps route parsing through a Web Worker so typing never blocks the UI thread.
• Validating HogQL/Hog before sending it to a backend - client code can parse-check a query or Hog program locally and surface a structured syntax error before round-tripping to a server.
• Sharing an AST with PostHog’s Python/Rust HogQL stack - because the grammar and AST shape are identical across hogql_parser (Python), @posthog/hogql-parser (WASM), and the Rust port, JS tooling can consume the same tree structure as the backend query planner.

Under The Hood

Architecture The package is a thin multi-target binding layer wrapped around a single ANTLR4-generated grammar core. HogQLLexer.cpp/HogQLParser.cpp (the largest generated files in the directory) implement tokenizing and parsing per the HogQL/Hog grammar; a shared visitor, HogQLParseTreeJSONConverter (parser_json.cpp, the largest hand-written file at roughly 3,400 lines), walks the ANTLR parse tree and emits a JSON AST via a small hand-rolled variant-based Json class (json.h/json.cpp) rather than pulling in a JSON dependency, keeping the embind/WASM binary lean. Three separate binding layers sit on top of that one converter: parser_python.cpp (a Python C extension, with parser_json_python.cpp for CPython-object construction), parser_wasm.cpp (Emscripten/embind, exporting parseExpr/parseSelect/parseOrderExpr/parseProgram/parseFullTemplateString/parseStringLiteralText), and a newer Rust rewrite path evidenced by cross-backend equivalence tests in the wider repo. The npm package specifically comes from CMakeLists.txt’s Emscripten branch, which compiles three WASM variants (ESM, browser-only ESM, CJS) all sharing the same parser_wasm.cpp entry point, wrapped by a small index.cjs shim. Errors are converted to structured JSON (buildWASMError) instead of thrown, and a HogQLErrorListener maps ANTLR’s line/column positions back to byte offsets, with a dedicated path for reporting an unprintable “unexpected character” token by its code point. In PostHog’s own frontend, the package is consumed exclusively inside a Web Worker with a small LRU-style memoization cache sitting in front of it, because a full-document parseSelect call costs an estimated 36ms per 1000 characters and would otherwise block the main thread on every keystroke in the SQL editor.

Tech Stack The core is C++20, built against a vendored ANTLR4 4.13.2 runtime pulled in via CMake’s FetchContent, producing several targets from one grammar: hogql_parser (a Python C extension distributed as prebuilt macOS/Linux, x86_64/arm64 wheels via cibuildwheel) and hogql_parser_wasm/_browser/_cjs (compiled with the Emscripten toolchain using -fwasm-exceptions and embind, single-file WASM with ALLOW_MEMORY_GROWTH). The npm package ships only the compiled WASM artifacts (dist/hogql_parser_wasm*.js, dist/index.cjs, dist/index.d.ts), produced by a pnpm + CMake/Ninja build. It lives inside PostHog’s much larger pnpm/Python/Rust monorepo, pinned as a plain version dependency in the frontend’s package.json rather than as a workspace package, and its release process has dedicated GitHub Actions workflows (separate npm and Python/Rust build pipelines) that gate on changes under common/hogql_parser/** and comment on a PR if its version wasn’t bumped.

Code Quality Testing is extensive but lives primarily alongside the shared grammar rather than in the npm package’s own directory: the wider repo’s HogQL test suite includes property-based tests and dedicated cross-backend equivalence tests, comparing output from the C++/ANTLR core against an in-progress Rust port for the same inputs. On the JS side specifically, two Jest test files exercise the compiled WASM bundle directly against real HogQL input. Error handling is deliberate and typed at the WASM boundary: parser_wasm.cpp never lets a C++ exception cross into JS — SyntaxError/NotImplementedError/ParsingError are all caught and serialized into a structured JSON object instead. Naming follows an explicit house convention documented in a CONTRIBUTING.md (snake_case throughout despite ANTLR’s Java-flavored camelCase generation, “auto” for verbose ANTLR types, explicit types elsewhere), and a .clang-format file enforces formatting on the hand-written C++. The public TypeScript types are hand-authored in index.d.ts rather than generated, and are fully documented with runnable JSDoc examples.

API Design The public surface is deliberately small and asynchronous-first: a single default export that resolves to an object exposing six methods, each returning a JSON string rather than a live object graph — a conscious trade against embind overhead for large parse trees, since a full parseSelect result can run many times the size of the input query text. The fallible-without-throwing convention (parse errors come back as JSON rather than a rejected promise or a thrown exception) lets a caller do one property check instead of a try/catch, and it behaves identically from three different runtimes (CPython, Node ESM/CJS, browser) off one shared C++ core, so a HogQL grammar change only has to land once. It isn’t inventing new parsing technology — it’s ANTLR4 plus a straightforward embind layer — but packaging one grammar as three separately-tuned build artifacts to dodge known Emscripten/bundler friction points (Node import.meta issues, browser-only environments, Jest’s CJS expectations) is more deliberate DX engineering than most single-target WASM packages bother with.