tree-sitter-elixir

tree-sitter-elixir is the official Elixir grammar for the Tree-sitter incremental parsing library, providing a complete syntax tree for Elixir source code including quoted expressions, sigils, heredocs, and Elixir’s macro-friendly AST semantics. It ships ready-to-use bindings for Rust, Node.js, Python, Go, and Swift, and is used in production by GitHub itself for Elixir syntax highlighting and code navigation.

The grammar is hand-written in grammar.js and compiled into a portable C parser, with a custom external scanner handling Elixir-specific lexing edge cases such as non-breaking newlines, unary-operator disambiguation, and quoted content across many delimiter types. Highlight, injection, and tag query files ship alongside the parser so editors and tools get syntax highlighting and symbol navigation out of the box.

What You Get

• A hand-maintained grammar.js definition covering the full Elixir syntax, including sigils, heredocs, quoted atoms, and the not in operator
• A generated C parser (src/parser.c) plus a custom external scanner (src/scanner.c) for lexing rules a context-free grammar alone can’t express
• Prebuilt language bindings for Rust, Node.js, Python, Go, and Swift so the grammar can be embedded directly in tools written in any of those languages
• highlights.scm, injections.scm, and tags.scm query files for out-of-the-box syntax highlighting, embedded-language injection, and symbol tagging
• A documented test corpus (test/corpus) covering expressions, terms, and edge-case syntax used to guard against regressions when the grammar changes

Common Use Cases

• Editor and IDE plugins use it to power Elixir syntax highlighting and code navigation, including GitHub’s own web UI
• Static analysis and linting tools use it to build a syntax tree for Elixir source without invoking the Elixir compiler
• Documentation generators and code-search tools use the tags query to extract function and module definitions for indexing
• Language tooling authors embed the Rust or Node bindings to add Elixir parsing support to their own developer tools

Under The Hood

Architecture A single grammar.js defines the precedence table and syntax rules; the tree-sitter-cli compiles it into a generated C parser (src/parser.c) that implements the LR/GLR parsing tables, while a hand-written external scanner (src/scanner.c) supplies context-sensitive tokens the grammar alone can’t express — quoted-content variants, non-breaking newlines, unary-operator disambiguation, not in, and quoted-atom starts — communicated through the TSLexer interface, with node-types.json documenting the resulting AST shape. The bindings/ directory then bridges that same C core into Rust (via a LanguageFn exposing the tree_sitter_elixir() symbol), Node (node-gyp-build/prebuildify), Python (setup.py), Go (cgo via go.mod), and Swift (Package.swift), so a change to the external scanner’s token set is the one point that ripples through every downstream binding and must be regenerated in lockstep.

Tech Stack The grammar itself is authored in JavaScript against the tree-sitter DSL, with tree-sitter-cli (^0.24.0) as the code generator; the Rust binding depends on tree-sitter-language 0.1.0 at runtime, tree-sitter 0.23.0 for dev/tests, and the cc crate to compile parser.c/scanner.c from bindings/rust/build.rs. The Node binding builds prebuilt per-platform binaries via node-addon-api and prebuildify; Python ships via setup.py/pyproject.toml; Go compiles the same C sources directly through cgo; Swift consumes it via SwiftPM. CI runs tree-sitter generate plus a WASM build on every push to main and auto-commits the regenerated parser and WASM artifact, with a separate tag-triggered workflow publishing all five ecosystem packages.

Code Quality Correctness is verified entirely through snapshot testing: test/corpus holds input/expected-parse-tree fixtures (expressions, terms, comments, unicode, edge syntax) run via tree-sitter test, plus separate test/highlight and test/tags suites that assert query correctness — an appropriate strategy for a grammar project, though scanner.c’s C logic has no dedicated unit tests beyond what the corpus exercises indirectly. Formatting is enforced via prettier (grammar.js) and clang-format (scanner.c) through an npm run format-check script, and the Rust binding carries a minimal smoke test confirming the grammar loads. No explicit type system beyond standard C89-style idioms in the scanner.

API Design The public surface is intentionally minimal and near-identical across bindings: a single exported LANGUAGE constant in Rust, one tree_sitter_elixir() FFI symbol underneath every other binding, so getting a working parser is a one-line call (parser.set_language(&language.into())) with essentially no boilerplate. The real engineering value isn’t a novel API surface — it’s grammar correctness for an unusually hard case: Elixir’s macro system means seemingly invalid code can be valid inside quote, and docs/parser.md documents in depth the specific lexer tricks (non-breaking newlines, unary-vs-binary disambiguation, Unicode identifier classes) required to match the real Elixir AST, which is why it’s already the parser GitHub uses in production.