bashlex

bashlex is a Python transliteration of the parser bash itself uses internally, reproducing its tokenizer and grammar without executing any of the resulting commands. It exposes a parse() function that turns a string of shell input into one or more AST trees built from typed node objects — command, pipeline, list, redirect, word, parameter, and process/command-substitution nodes — each carrying source position spans so callers can splice or rewrite parts of the original text.

Because it mirrors bash’s own C parser closely (comments reference bash’s parse.y source line by line), bashlex understands constructs a naive shlex-based approach misses, such as nested command and process substitutions, redirections, and heredocs. It also ships a lighter-weight split() helper for shlex-style tokenization when a full AST isn’t needed. The project originated as the parsing backend for explainshell.com and is now used by tools that need to statically analyze, rewrite, or lint shell scripts.

What You Get

• A parse() function that returns a list of AST trees for arbitrary bash input, including multi-statement scripts.
• A typed AST (bashlex.ast.node) with distinct kinds — command, pipeline, list, redirect, word, parameter, tilde, heredoc — each carrying a pos source span for exact substring locations.
• A nodevisitor base class for walking the tree with per-kind visit* callback methods, so consumers can implement rewriters or analyzers without hand-rolling traversal.
• A standalone split() tokenizer for shlex-style word splitting that still understands command/process substitution, unlike the stdlib shlex module.

Common Use Cases

• Building linters or static analyzers for shell scripts that need real syntax awareness rather than regex matching.
• Writing source-to-source shell rewriters (e.g. replacing command substitutions or redacting arguments) using exact position spans from the AST.
• Tokenizing complex shell command lines for embedding in another language or tool, where shlex.split breaks on substitutions.
• Powering explainshell.com-style tools that annotate or explain each part of a shell command.

Under The Hood

Architecture bashlex separates concerns cleanly across five modules: tokenizer.py implements a hand-written lexer (~1,100 lines) that mirrors bash’s own character-class tables (sh_syntaxtab) and token types; yacc.py is a vendored LALR(1) parser-generator runtime (a PLY-style yacc) that parser.py’s grammar rules (p_* functions, e.g. p_inputunit, p_redirection, p_word_list) drive to build a tree of ast.node objects; subst.py and heredoc.py perform post-lex processing for word/parameter expansion and heredoc bodies; and state.py plus flags.py carry cross-module parser state (e.g. CMDSUBST, EOFTOKEN flags) between the tokenizer and grammar layers. This tokenizer-then-grammar-then-AST pipeline closely tracks bash’s actual C implementation, so the abstraction that would break the most if changed is the shared node representation in ast.py, since every grammar rule constructs nodes by kind and every consumer (including the bundled nodevisitor) dispatches on that same kind string.

Tech Stack bashlex is pure Python with no runtime dependencies beyond the standard library (re, collections, enum) and a conditional enum34 backport for pre-3.4 interpreters; there is no web framework, ORM, or database layer since the project’s sole surface area is a parsing API. Packaging uses a plain setuptools setup.py plus a minimal pyproject.toml declaring the build backend, and CI (GitHub Actions) runs the test suite via make tests across several recent Python versions on Ubuntu, macOS, and Windows, with requirements.txt (build, twine, pytest) driving the install rather than a lockfile-based tool.

Code Quality Testing lives in tests/test_parser.py and tests/test_tokenizer.py, built around per-node-kind fixture constructors (commandnode, wordnode, pipelinenode, etc.) that make expected ASTs easy to express and compare via ast.node’s structural equality; setup.cfg also enables pytest’s doctest-modules option so any doctest-style examples in the library are exercised. Error handling favors explicit, typed failures — a dedicated errors module defines exception classes, and grammar-rule helpers raise NotImplementedError/AssertionError with contextual token info rather than swallowing failures. There is no type-hinting, dataclasses, or linter/formatter configuration anywhere in the repo, and node objects are built from keyword arguments into a plain dict rather than a typed schema, trading static safety for flexibility across many node kinds.

API Design The public surface is intentionally small: parse(), parsesingle(), and split() cover the two common needs — full AST vs. shlex-style tokenization — without asking callers to instantiate parser classes directly. ast.node’s custom repr and dump() print a readable, indented tree during debugging, and the nodevisitor base class turns tree-walking into overriding a handful of visit<kind> methods with return-value-based recursion control, which keeps rewriter/analyzer code (as in the bundled command-substitution-remover example) short. The tradeoff is that nodes are dynamically-shaped objects rather than a typed union, so IDE autocompletion and static type-checking of node attributes isn’t available, and callers must consult the README or a dumped tree to learn which attributes a given kind carries.