tree-sitter (Python)

tree-sitter is the official Python binding for the Tree-sitter parsing system, exposing a fast C core through an ergonomic Python API. It parses source code in dozens of programming languages into concrete syntax trees, then incrementally re-parses only the parts that change as you edit, making it fast enough to run on every keystroke.

The package ships pre-compiled wheels with no external dependencies, and works with any Tree-sitter grammar installed as a separate language package. It powers syntax highlighting, code navigation, structural search, linting, and static analysis in editors and developer tools.

What You Get

• A Parser that turns bytes of source code into a concrete syntax Tree for any installed grammar
• Typed Node and TreeCursor APIs for efficiently traversing and inspecting the syntax tree
• A Query/QueryCursor engine for pattern-matching over trees with capture and predicate support
• Incremental re-parsing via Tree.edit and changed_ranges for fast updates on edited source
• Pre-compiled, dependency-free wheels for major platforms with bundled type stubs (py.typed)

Common Use Cases

• Syntax highlighting and structural code navigation in editors and IDE tooling
• Building linters, refactoring tools, and static analysis that operate on syntax trees
• Extracting structure (functions, classes, imports) from source across many languages

Under The Hood

Architecture - The package is a thin Python layer over the Tree-sitter C library, which is vendored as a git submodule at tree_sitter/core (pinned to the release-0.26 branch). The public surface lives in tree_sitter/__init__.py, which re-exports C extension types (Language, Parser, Node, Tree, TreeCursor, Query, QueryCursor, Range, Point, LookaheadIterator) from the compiled _binding module. Each type is implemented in its own C translation unit under tree_sitter/binding/ (parser.c, node.c, tree.c, query.c, query_cursor.c, tree_cursor.c, language.c, etc.), with module.c wiring the CPython module together and managing per-module state such as a shared default cursor. Parsing produces a concrete syntax tree that can be incrementally edited and re-parsed, so only changed ranges are recomputed.

Tech Stack - The core is C11 (about 69% of the codebase) with a Python layer (about 31%) targeting Python 3.10+. It builds via setuptools with a custom build_ext in setup.py that compiles the C sources plus the vendored core/lib/src/lib.c, setting flags like -std=c11 -fvisibility=hidden (and MSVC equivalents) and defining macros including PY_SSIZE_T_CLEAN and the Tree-sitter version. Distribution uses cibuildwheel to produce platform wheels; there are no runtime library dependencies.

Code Quality - The project ships full type stubs (__init__.pyi, py.typed) and a comprehensive unittest suite (roughly 1,700 lines across test_parser, test_node, test_tree, test_query, test_language, and test_lookahead_iterator) that exercises real grammars (HTML, JavaScript, JSON, Python, Rust) pulled in as test extras. Ruff is configured for linting and formatting, mypy for type checking, and clang-format plus rstcheck enforce C and docs style. CI runs on GitHub Actions.

API Design - The API is compact and Pythonic: construct a Language from a grammar, pass it to Parser, and call parse() with bytes or a read callable supporting byte-offset or point-based reads and UTF-8/UTF-16 encodings. Node access uses readable methods like child_by_field_name and start_point/end_point, tree walking uses an explicit TreeCursor, and queries use an s-expression pattern language with captures and predicates. Sphinx-generated documentation and runnable examples (examples/usage.py, examples/walk_tree.py) keep the learning curve low for anyone familiar with the Tree-sitter model.