tree-sitter-kotlin

tree-sitter-kotlin is a Tree-sitter grammar for the Kotlin language, translating the official Kotlin grammar reference into an incremental, error-tolerant parser suitable for editors, linters, and code-intelligence tooling. The grammar is authored once in grammar.js and compiled into a shared C parser core, then wrapped with idiomatic bindings for C, Go, Node.js, Python, Rust, and Swift, plus a WebAssembly build with an interactive browser playground.

What sets this grammar apart is its JetBrains cross-validation pipeline: real Kotlin source fixtures and expected parse trees are vendored from the official Kotlin compiler’s own test suite at a pinned commit, then structurally diffed against tree-sitter-kotlin’s output across 110+ node-type mappings to measure fidelity against the canonical parser (currently 61.2% structural match on clean parses), with outstanding mismatches tracked and categorized for incremental fixing.

What You Get

• A complete Tree-sitter grammar for Kotlin (.kt/.kts) covering classes, coroutines, string templates, and the rest of the language surface
• Native bindings for C, Go, Node.js, Python, Rust, and Swift built from one shared grammar source
• Prebuilt syntax-highlighting (highlights.scm) and code-navigation (tags.scm) query files ready for editor integration
• A WebAssembly build plus a browser-based playground for interactively exploring parse trees
• A JetBrains PSI cross-validation harness that measures and tracks grammar correctness against the official Kotlin compiler’s reference parser
• An external scanner handling Kotlin’s context-sensitive lexical features — automatic semicolon insertion, nested comments, and interpolated triple-quoted strings

Common Use Cases

• Editor syntax highlighting - editors and IDEs use the bundled highlights.scm query to color Kotlin source as you type
• Code navigation tooling - language servers and code-intelligence tools use tags.scm to power go-to-definition and symbol search
• Static analysis and linting - linters and refactoring tools parse Kotlin into a concrete syntax tree without needing a full Kotlin compiler
• Cross-language embedding - projects needing to parse Kotlin from Rust, Go, Python, Node.js, C, or Swift pull in the matching binding rather than shelling out to kotlinc

Under The Hood

Architecture The project separates a JS-authored grammar definition from a machine-generated parser. grammar.js declares precedence classes and production rules translating the official Kotlin grammar reference into a tree-sitter grammar; tree-sitter generate compiles this into a C state machine (src/parser.c, explicitly marked generated/do-not-edit in CONTRIBUTING.md) plus src/grammar.json and src/node-types.json describing node shapes. A hand-written external scanner (src/scanner.c) supplements the generated lexer for context-sensitive tokens the LR core can’t express directly — automatic semicolon insertion, nested comments, and interpolated triple-quoted string delimiters — explicitly modeled on tree-sitter-javascript’s and Julia’s scanners per the file’s own comments. Six language bindings (bindings/{c,go,node,python,rust,swift}) each wrap the same generated parser.c/scanner.c pair with idiomatic glue, so the core grammar logic lives in exactly one place and every binding is a thin adapter. Downstream consumption is query-driven: queries/highlights.scm and queries/tags.scm are separate declarative files consumers run against the resulting tree, decoupling grammar internals from highlighting/navigation concerns. A tools/cross-validation/ subsystem independently fetches JetBrains’ own PSI reference-parser test fixtures at a pinned commit and structurally diffs tree-sitter’s parse trees against them, acting as an external correctness oracle decoupled from the corpus tests in test/corpus/.

Tech Stack The grammar is authored in JavaScript (grammar.js, using tree-sitter’s DSL of seq/choice/prec/token combinators) and compiled by tree-sitter-cli (^0.24.4, devDependency) into C. The Node binding depends on node-addon-api ^8.2.2 and node-gyp-build ^4.8.2, built via binding.gyp/node-gyp, with prebuildify ^6.0.0 generating prebuilt binaries; it declares an optional peer dependency on tree-sitter ^0.21.1. The Rust binding (Cargo.toml) depends on tree-sitter-language 0.1, builds the C sources via a build.rs using the cc 1.1 crate, and dev-depends on tree-sitter 0.24 for its doctest. The Go (go.mod), Python (pyproject.toml/setup.py), and Swift (Package.swift) bindings each point back at the same C sources. WebAssembly builds go through Emscripten, feeding a browser playground deployed to fwcd.github.io/tree-sitter-kotlin. Cross-validation tooling uses vitest ^4.1.0. CI runs on GitHub Actions. There is no runtime dependency beyond the tree-sitter ecosystem itself — a narrowly-scoped, single-purpose grammar multiplexed across six language ecosystems from one source of truth.

Code Quality Testing relies on tree-sitter’s own corpus format — test/corpus/ contains hand-written .txt files pairing Kotlin snippets with expected parse trees, run via tree-sitter test. This is supplemented by an unusual second tier: tools/vendor-jetbrains-tests.js pulls real-world Kotlin source from the JetBrains/kotlin compiler’s own test fixtures and auto-generates additional corpus cases from files that parse cleanly, while tools/cross-validation/main.js structurally diffs the resulting trees against JetBrains’ PSI reference output across a broad set of node-type mappings — a genuine external-oracle testing strategy beyond typical grammar-only unit tests. Known mismatches are tracked explicitly rather than hidden: tools/cross-validation/excluded.txt lists excluded fixtures and TODO.md categorizes outstanding grammar issues by difficulty, an auditable trail of what’s not yet correct. The generated src/parser.c versus the hand-written src/scanner.c are explicitly documented in CONTRIBUTING.md as generated-do-not-edit versus hand-maintained, good hygiene for a codebase where most of the “code” is machine output. CI runs on GitHub Actions on every push. There is no traditional static type system (grammar.js is untyped JS, scanner.c is C), but correctness is enforced structurally through dual corpus-plus-cross-validation testing rather than compile-time types.

API Design Each binding exposes an idiomatic, minimal surface for its ecosystem — for example the Rust crate’s bindings/rust/lib.rs exposes a single LANGUAGE constant (a LanguageFn) plus re-exported NODE_TYPES and HIGHLIGHTS_QUERY string constants, requiring only parser.set_language(&LANGUAGE.into()) to get started, with a working doctest embedded directly in the crate docs. The Node package mirrors this pattern via prebuilt native bindings loaded through node-gyp-build, so consumers install and require the package without a local compiler toolchain in the common case. Because every binding wraps the same generated core, the mental model transfers directly across languages: load the language object, hand it to a tree-sitter Parser, and consume the resulting tree with the shared node-types.json shape and the bundled highlights.scm/tags.scm queries — boilerplate is close to zero once the ecosystem-appropriate package is installed.