rustpython-ast

rustpython-ast provides the Rust type definitions for Python’s abstract syntax tree, mechanically generated from a Python.asdl grammar file using the same ASDL format CPython itself relies on. Originally built as the core AST layer for the RustPython interpreter, the crate was split out as a standalone, well-packaged library so other Rust-based Python tooling — including early versions of the Ruff linter — could depend on a shared, grammar-accurate AST representation instead of each maintaining its own.

Every node type is generic over a source-range parameter, so the same generated structs work both as a lightweight, allocation-free AST (EmptyRange) and as a fully location-tracked tree (TextRange) depending on which optional features a consumer enables. Additional feature flags layer in a Fold trait for AST-rewriting passes, a Visitor trait for read-only traversal, constant-folding optimization, and Python source unparsing — letting downstream crates opt into exactly the capabilities they need.

What You Get

• Grammar-generated AST types — Stmt, Expr, Pattern and related node structs generated directly from Python.asdl, mirroring CPython’s own ast module naming.
• Optional source-location tracking — enable the location feature to get full TextRange spans on every node, or leave it off for a zero-cost, allocation-free tree.
• Fold and Visitor traits — feature-gated fold and visitor traits for writing AST-rewriting passes or read-only traversals without hand-rolling dispatch.
• Python source unparsing — the unparse feature turns an AST back into Python source text, useful for codemods and formatters.
• Constant-folding optimizer — the constant-optimization feature exposes a ConstantOptimizer that folds literal expressions at the AST level.

Common Use Cases

• Building a Python linter or static analyzer that needs a typed, traversable AST instead of parsing text with regex.
• Writing a Python-to-Python codemod or auto-formatter that parses source, transforms the tree via Fold, and unparses it back.
• Embedding a Python front-end in a Rust interpreter or transpiler that needs the same AST shape CPython produces.
• Building editor tooling (language servers, syntax highlighters) that needs precise source-range spans per AST node.

Under The Hood

Architecture The crate is generated from Python.asdl via asdl_rs.py (a Python code generator that reads the ASDL grammar and emits Rust type definitions into src/gen/). generic.rs defines the core Rust structs and enums for each AST node type (Stmt, Expr, Pattern, and more) generic over a range type R, defaulting to TextRange. Optional features layer additional capability on top: location adds source-location tracking via located.rs and source_locator.rs, fold adds an AST-rewriting Fold trait in fold.rs, visitor generates a read-only Visitor trait, unparse converts the tree back to Python source in unparse.rs, and constant-optimization folds literal expressions via optimizer.rs. Hand-written builtin.rs and impls.rs add small ergonomic helpers — like CmpOp::as_str() and Expr::python_name() — on top of the generated types. Because nodes are generic over R, the same generated struct definitions serve both a lightweight EmptyRange (default, zero-cost) and a full TextRange variant, letting downstream consumers such as the parser crate, the RustPython interpreter, and Ruff opt into whichever precision they need without maintaining two parallel type hierarchies.

Tech Stack Rust 2021 edition workspace member (rust-version 1.72.1). Uses is-macro for discriminant helper macros, optional num-bigint or malachite-bigint for big-integer constant representation, static_assertions for compile-time invariant checks, and workspace-internal rustpython-parser-core for shared TextSize and source-location primitives plus rustpython-literal (feature-gated) for stringifying literals during unparse. No runtime dependencies beyond these — it is a pure data-definition crate with zero I/O and no async runtime. Code generation is driven by a Python script (asdl_rs.py, using a stdlib-only asdl.py parser) run offline against Python.asdl to produce src/gen/*.rs, checked into the repo rather than generated at build time via build.rs. CI runs cargo test, clippy, and rustfmt across the whole workspace with multiple feature-flag combinations on Ubuntu and Windows, plus ruff linting the Python generator script and a spell-checker pass.

Code Quality The ast crate itself has no dedicated test modules — a static_assertions::assert_eq_size! size check is left commented out as a TODO in impls.rs — so correctness is validated indirectly through the sibling parser crate’s extensive insta snapshot-test suite, which round-trips real Python source through this AST across statements, expressions, f-strings, pattern matching, and comprehensions. Error handling is minimal by design since the crate defines data rather than fallible operations; the one hand-written behavior, Expr::python_name(), is an exhaustive match with no default arm, so the compiler enforces coverage as new AST variants are added. Naming closely mirrors CPython’s own ast module and the ASDL grammar, aiding cross-reference with Python’s documentation. clippy and rustfmt are enforced in CI with warnings denied, and multiple feature-flag combinations are exercised in the test matrix, though most quality risk lives upstream in the code generator rather than in the generated output itself.

API Design The public API favors mirroring CPython’s ast module field-for-field, so anyone familiar with Python’s own AST can read the generated Rust types with little translation. The generic range parameter is the main ergonomic trade-off: it keeps the crate zero-cost for consumers that don’t need spans, but it does mean every node type carries a type parameter that shows up throughout downstream code. Documentation is thin at the crate level — the README describes the project’s origin and sibling projects rather than showing usage, and there are no dedicated examples — so a newcomer’s fastest path to understanding the API is reading the generated gen/ modules and the sibling parser crate’s tests rather than prose docs.