ANTLR4 Python3 Runtime

The ANTLR4 Python3 runtime is the companion library that parsers and lexers generated by the ANTLR tool import when they run. ANTLR (ANother Tool for Language Recognition) turns a grammar into recognizer code; this package provides the Python 3 implementation of the shared machinery that generated code relies on - input streams, token streams, the lexer and parser base classes, the adaptive LL(*) prediction engine, and parse-tree structures with listeners and visitors.

By pairing generated recognizers with this runtime, developers can tokenize and parse any structured text - programming languages, config formats, expressions, or data files - and then walk the resulting parse tree to interpret, transform, or validate it. The runtime mirrors ANTLR’s other language targets so the same grammar produces equivalent parsers across Python, Java, C#, Go, and more.

What You Get

• Input and token stream classes (InputStream, FileStream, CommonTokenStream)
• Base Lexer, Parser, and Recognizer classes for generated code
• The adaptive LL(*) prediction engine (ATN, DFA, prediction contexts)
• Parse-tree nodes with listener and visitor walking
• A TokenStreamRewriter for source-to-source transformations
• Cross-target parity with ANTLR’s Java, C#, Go, and other runtimes

Common Use Cases

• Running an ANTLR-generated parser for a custom language or DSL in Python
• Parsing configuration or expression grammars and evaluating them
• Walking parse trees with visitors to interpret or transform input
• Building linters, transpilers, or analyzers on top of a grammar

Under The Hood

Architecture - The runtime lives under runtime/Python3/src/antlr4. Input flows through stream classes (InputStream, FileStream, CommonTokenStream) into a generated Lexer and Parser, both of which extend the runtime’s Recognizer base. The heart is the ALL(*) prediction machinery in atn/ (augmented transition network), dfa/ (cached decision automata), and PredictionContext.py, which lets parsers make adaptive parsing decisions. Recognized input becomes a ParserRuleContext tree in tree/, walkable via listeners and visitors, with TokenStreamRewriter.py enabling in-place source edits.

Tech Stack - Pure Python 3 packaged with a modern pyproject.toml under the antlr4 import namespace. It has no runtime dependencies and no compiled extensions - the entire prediction engine is implemented in Python - which keeps installation trivial across platforms. It is one target of the larger polyglot ANTLR monorepo whose tool is written in Java.

Code Quality - The Python3 runtime ships its own tests/ directory exercising streams (TestFileStream, TestInputStream), interval sets, and generated parser/lexer behavior against sample grammars in tests/expr and tests/parser. Because ANTLR maintains parity across many language targets, the runtime’s structure closely mirrors the reference implementation, which keeps behavior consistent and well specified.

API Design - Application authors rarely touch the runtime directly; they use the ANTLR tool to generate parser and lexer subclasses and then instantiate them with a stream, call the start rule, and walk the returned tree. That indirection makes day-to-day use clean, but understanding the runtime internals (ATN, prediction contexts) has a steep learning curve tied to ANTLR’s parsing theory. The bundled README and the broader ANTLR documentation cover the generated-parser workflow.