parso

Parso is a pure-Python parser for Python source code that supports error recovery, meaning it returns a usable syntax tree even for invalid or incomplete code instead of just raising an exception. It ships grammar definitions for every Python version from 3.6 through 3.14, so a single parso installation can parse code targeting a different Python version than the one it’s running under, a common need for editors and linters that support multiple environments.

Originally extracted from the Jedi autocompletion library so it could be reused elsewhere, parso combines a classic pgen2-based grammar and tokenizer pipeline with a diff parser that reuses unchanged parts of a previously parsed tree, making repeated re-parses of a large, mostly-unchanged file fast enough for interactive use in an editor. On top of the parse tree it layers a normalizer framework used to surface multiple syntax errors and PEP 8 style issues in one pass.

What You Get

• Multi-version grammars - bundled BNF grammar files for Python 3.6 through 3.14 let you parse code for a version other than the one running parso.
• Error-recovering parser - invalid or incomplete code still returns a usable tree with error nodes instead of raising immediately.
• Incremental diff parser - DiffParser reuses unchanged subtrees between edits, avoiding a full re-parse on every keystroke.
• Multi-error reporting - Grammar.iter_errors() surfaces every syntax error in a module in one pass, not just the first.
• Typed public API - ships a py.typed marker and .pyi stubs for static-analysis-friendly consumption.

Common Use Cases

• Building an IDE or editor plugin - power autocompletion, go-to-definition, and live error squiggles the way Jedi does, using the diff parser to stay responsive on large files.
• Writing a linter or style checker - walk the syntax tree with a custom Normalizer to flag project-specific style or correctness issues, the same mechanism parso’s own PEP 8 checker uses.
• Parsing code you don’t control the Python version of - target a specific version="3.9" grammar regardless of which interpreter parso itself runs under, useful for multi-version codebases or migration tooling.
• Refactoring tools - use the concrete syntax tree, which preserves whitespace and comments, to make source-preserving code transformations instead of round-tripping through an AST that loses formatting.

Under The Hood

Architecture The Grammar class in parso/grammar.py orchestrates the pipeline: source is tokenized (parso/python/tokenize.py) against a pgen2-generated grammar built from bundled BNF text files (parso/pgen2/generator.py, one grammar file per supported Python version), producing a concrete syntax tree defined in parso/tree.py and specialized for Python in parso/python/tree.py. The standout piece is the DiffParser (parso/python/diff.py), which diffs new source against a previously parsed tree and reuses unchanged subtrees instead of reparsing from scratch, the feature that lets IDEs like Jedi stay responsive on large files after small edits. An on-disk cache (parso/cache.py) persists parsed modules keyed by content hash and grammar version. Error detection and style checking are layered on top through a normalizer/visitor pattern (parso/normalizer.py, parso/python/errors.py, parso/python/pep8.py) that walks the finished tree rather than being baked into the parser itself, so the core parse/tokenize path stays independent of any particular error-reporting policy.

Tech Stack Pure Python with zero runtime dependencies, supporting Python 3.6+ and tested across 3.8 through 3.13 in a GitHub Actions matrix. Packaging uses classic setuptools/find_packages via setup.py and setup.cfg. Optional extras separate concerns cleanly: testing pulls in pytest and docopt, qa pulls in flake8 and zuban (a fast Rust-based type checker) plus type stubs for setuptools. Documentation is built with Sphinx and published on Read the Docs.

Code Quality Twenty test files cover the tokenizer, the pgen2 grammar generator, the diff parser (including a dedicated fuzzer for the incremental-parsing path), PEP 8 checks, and syntax-error recovery, backed by a large fixture corpus of known-bad syntax examples for regression coverage. The public API is fully typed and enforced in CI by both flake8 and zuban with strict settings (subclassing-any disallowed, unreachable code and redundant casts flagged), and a separate coverage job runs alongside the version matrix. Error handling favors explicit exception types over silent failures, and the diff parser includes sanity checks that fall back to a full reparse whenever the incremental result can’t be trusted.

API Design The public surface is deliberately small: parso.parse(code, version=...) is a one-line entry point that needs no setup, while parso.load_grammar() exposes the fuller Grammar object with caching and diff-caching flags for advanced callers. Every public method carries a docstring with runnable doctest examples, mirrored in dedicated Sphinx usage and parser-tree reference pages. Because parso was extracted from Jedi to be reused elsewhere, its ergonomics were already proven by a real downstream consumer before being published standalone, version selection is a single string keyword rather than a config object. The tree API itself trades casual-scripting friendliness for completeness, exposing the full grammar and token detail that tools like Jedi and autopep8 actually need.