tree-sitter

tree-sitter is a parser-generator tool and incremental parsing library that builds concrete syntax trees for source files and can efficiently re-parse just the edited portion of a file as it changes, keeping parses fast enough for keystroke-level use in editors. The Rust crate wraps the core C parsing runtime, and grammars for individual languages are distributed as separate crates (tree-sitter-language and per-language grammar crates) that plug into this shared runtime, with web-tree-sitter providing a WASM build for browser and Node use.

It is the parsing engine behind syntax highlighting, code folding, and structural editing in tools like Neovim, GitHub’s code viewer, and Zed, and its language-agnostic grammar model has made it a de facto standard for building editor tooling that needs error-tolerant, incremental parsing.

What You Get

• A Rust Parser/Tree/Node API for parsing source text into a concrete syntax tree
• Incremental re-parsing that only reprocesses the edited region of a document
• A query language (S-expression based) for pattern-matching over syntax trees
• A tree-sitter CLI (crates/cli) for generating, testing, and highlighting with grammars
• WASM bindings (web-tree-sitter) for running grammars in the browser or Node

Common Use Cases

• Powering syntax highlighting and code folding in an editor or IDE plugin
• Building structural/AST-aware code search, linting, or refactoring tools
• Implementing ‘jump to definition’ or symbol indexing features via tree queries
• Parsing configuration or markup formats where error recovery matters (e.g. editing mid-typing)

Under The Hood

Architecture — The workspace separates concerns cleanly: lib/src holds the core C parsing runtime (lexer, GLR parser, incremental tree diffing, Unicode handling under src/unicode), lib/binding_rust exposes this runtime as safe Rust types (Parser, Tree, Node, Query), and lib/binding_web compiles the same core to WebAssembly; crates/generate implements the grammar-to-parser-table compiler, crates/highlight/crates/tags provide query-based highlighting/symbol extraction, and crates/cli ties these into the tree-sitter command-line tool. Tech Stack — A hybrid C/Rust codebase (C for the portable parsing core, Rust for bindings, CLI, and grammar tooling) built with Cargo workspaces plus optional Zig (build.zig) and CMake (CMakeLists.txt) build paths, targeting Rust edition 2024 with MSRV 1.90, and using clippy’s nursery/cargo lint groups workspace-wide. Code Quality — The project has an extensive test/ suite (grammar corpus tests, highlight tests, query tests) alongside per-crate unit tests, strict workspace-level Clippy lint configuration (denying dbg! macros, warning on nursery/cargo lints), and active CI across multiple build systems (Cargo, Zig, CMake, Swift Package Manager), reflecting its role as foundational infrastructure many tools depend on. API Design — The Rust API centers on a small set of composable types (Parser::parse, Tree::root_node, Node::child/walk, Query::matches) that map directly onto the underlying C API, keeping the FFI boundary thin; the separate query language lets consumers express highlighting/refactoring rules declaratively instead of hand-writing tree-walking code, which is a major reason it has become a shared substrate across many unrelated editor and tooling projects.