asttokens

asttokens takes a Python abstract syntax tree (from the standard ast module or astroid) and augments each node with the exact range of source text and tokens it corresponds to, something Python’s built-in AST does not provide out of the box for every node type. This lets tools recover the precise original source snippet, line/column span, or surrounding whitespace and comments for any expression, statement, or function definition in a parsed file.

Because of this, asttokens has become a foundational dependency for source-aware Python tooling — IDEs, linters, code-formatters, coverage tools, and interactive debuggers rely on it (directly or via the executing and stack_data libraries) to map runtime frames and lint findings back to exact source locations.

What You Get

• ASTTokens class that annotates an entire module’s AST with source-text spans in one pass
• Per-node .first_token / .last_token and .get_text(node) accessors for exact source recovery
• Support for both the standard library ast module and the astroid library used by pylint
• Token-range utilities for building precise source-to-source refactoring tools
• Typed public API (py.typed) for static-analysis-friendly consumption
• Works across supported CPython versions with tokenizer-compatibility shims

Common Use Cases

• Building linters or static-analysis tools that need to report exact source snippets for a finding
• Powering source-mapping in interactive debuggers and REPLs (as used by IPython’s stack_data/executing)
• Implementing code-refactoring or code-generation tools that rewrite specific AST nodes in place
• Extracting exact function/class source text for documentation generators or coverage reporting

Under The Hood

Architecture - the core ASTTokens class (in asttokens.py, ~434 lines) tokenizes the source with Python’s tokenize module, then mark_tokens.py (~489 lines) walks the AST recursively and assigns each node a first_token/last_token pair by reconciling node line/column offsets against the token stream, handling edge cases the stdlib ast module leaves ambiguous (implicit string concatenation, parenthesized expressions, multi-line statements); line_numbers.py provides byte-offset-to-line/column translation, and astroid_compat.py adapts the same logic to astroid’s node model for pylint-style consumers. Tech Stack - pure Python with setuptools_scm for git-tag-based versioning, optional astroid support (imported lazily to avoid a hard dependency), and no other runtime dependencies; the project has a mypy-typed public surface via py.typed. Code Quality - the test suite spans multiple targeted files (test_mark_tokens.py, test_line_numbers.py, test_tokenless.py, test_astroid.py) covering edge cases across Python grammar versions and both AST backends, with tox.ini running the matrix across supported interpreter versions and .pylintrc/mypy config enforcing static-analysis cleanliness on the codebase itself. API Design - the public surface centers on a single ASTTokens(source, tree=...) constructor plus .get_text(node), keeping the mental model simple (one object annotates one module) while supporting both eager AST parsing and pre-parsed trees, which is why it has become a stable base dependency other tools build on rather than a user-facing framework.