tree-sitter-python

tree-sitter-python is the official Python language grammar for tree-sitter, the incremental parsing library used by editors and developer tools to build concrete syntax trees. It defines Python’s syntax (indentation-based blocks, comprehensions, decorators, match statements, type hints) in a declarative grammar that compiles to a generated C parser plus bindings for Rust, Node.js, Python, Go, and Swift.

Editors, linters, and static-analysis tools consume this grammar to parse Python source incrementally and error-tolerantly, powering syntax highlighting, code folding, and structural navigation without a full CPython front-end.

What You Get

• A complete, actively maintained Python grammar covering modern syntax including structural pattern matching and type hints
• A generated, dependency-free C parser (parser.c) plus a hand-written external scanner (scanner.c) for indentation-sensitive tokens
• Official bindings for Rust, Go, Node.js, Python, and Swift for embedding the grammar directly in tooling
• Highlight and tag queries (queries/highlights.scm, queries/tags.scm) mapping syntax nodes to editor categories
• A broad corpus of test fixtures (test/corpus, test/highlight, test/tags) validating grammar correctness against real Python code

Common Use Cases

• Editor syntax highlighting - power tree-sitter-based highlighting for Python files in editors like Neovim, Helix, and Zed
• Code navigation tooling - build ‘jump to definition’ and structural outlines by querying the parsed Python syntax tree
• Static analysis and linters - traverse the concrete syntax tree to implement custom Python lint rules without importing the ast module
• Language server implementations - use the grammar as the incremental parsing layer underneath a Python LSP server

Under The Hood

Architecture - The grammar is authored in grammar.js as a JavaScript DSL describing Python’s syntax rules; because Python’s block structure is indentation-based rather than delimiter-based, the grammar pairs with a hand-written external scanner (src/scanner.c) that emits synthetic INDENT/DEDENT/NEWLINE tokens the declarative grammar rules can then consume, alongside a table-driven src/parser.c generated by the tree-sitter CLI.

Tech Stack - The grammar source is JavaScript, but the shipped artifact is generated C code with zero runtime dependencies beyond tree-sitter-language; the repo ships first-class bindings for Rust (Cargo.toml), Go (go.mod), Node.js (package.json, binding.gyp), Python (pyproject.toml, setup.py), and Swift (Package.swift).

Code Quality - Correctness is validated through test/corpus (parse-tree fixtures), test/highlight (highlight-query fixtures), and test/tags (symbol-tagging fixtures) — a notably broader test surface than most tree-sitter grammars, reflecting Python’s popularity and the complexity of its indentation-sensitive scanner; CI runs across all bindings before each release.

API Design - Consumers interact through a minimal surface: a single exported language constant handed to the generic tree-sitter runtime API, so there’s effectively no Python-specific API to learn beyond tree-sitter’s own parser/query interface, keeping integration overhead low.