mistune

Mistune is a Python Markdown parser built for speed and extensibility. It parses text into an intermediate token tree in two phases — block-level then inline-level — before handing that tree to a renderer, so the same parse can be turned into HTML, an AST (as plain dicts), Markdown, or reStructuredText without re-parsing the source.

The library ships a strict CommonMark 0.31.2-compliant core plus an opt-in plugin system covering GitHub-flavored extras (tables, strikethrough, task lists, footnotes, URL auto-linking), math ($...$ and $$...$$), definition lists, abbreviations, ruby annotations, and a directive framework (fenced and RST-style) for admonitions, images/figures, table-of-contents blocks, and file includes. It is the Markdown engine behind projects like Flask and mkdocs-derived tooling, and its speedup plugin swaps in a Cython-accelerated inline scanner when available.

What You Get

• A CommonMark 0.31.2-compliant block and inline parser producing a typed token tree
• Swappable renderers: HTMLRenderer, an AST (list-of-dicts) mode, MarkdownRenderer, and RSTRenderer
• Built-in plugins for tables, strikethrough, footnotes, task lists, definition lists, abbreviations, math, ruby text, spoilers, and URL auto-linking
• A directive framework (FencedDirective, RSTDirective) for admonitions, images/figures, table-of-contents blocks, and file includes
• Before-parse, before-render, and after-render hook points for customizing the pipeline without subclassing the parser
• A small CLI (mistune command) for converting files from the shell
• Full type annotations checked under mypy --strict

Common Use Cases

• Rendering user-submitted Markdown (comments, issues, docs) to safe HTML with escaping and protocol allow-listing
• Building a static-site or documentation generator that needs a table-of-contents hook and custom heading IDs
• Producing a structured AST from Markdown for a custom renderer (e.g. PDF export or a non-HTML target)
• Extending Markdown syntax with custom directives (callouts, embeds, includes) for an internal docs tool
• Converting between Markdown and reStructuredText via the built-in renderers
• Embedding math notation (LaTeX-style $...$) in rendered documentation pages

Under The Hood

Architecture Mistune parses in two passes: BlockParser (block_parser.py) walks the source line-by-line into a flat list of block tokens (paragraphs, headings, lists, fenced code, etc.), storing raw text spans without descending into inline syntax; Markdown.parse() (markdown.py) then calls _iter_render, which recursively hands each block’s text to InlineParser (inline_parser.py) to resolve emphasis, links, and code spans into a children token list. The resulting token tree is state carried in a BlockState/InlineState pair (core.py) rather than mutated globally, and is handed to whichever BaseRenderer subclass is configured (or returned as raw dicts when renderer=None), so the same parse output can drive HTML, Markdown, or RST rendering without re-parsing. Extension points are hook lists (before_parse_hooks, before_render_hooks, after_render_hooks) that plugins register on a Markdown instance via plugin(self), keeping the parser class itself closed to plugin-specific logic — for example toc.py’s add_toc_hook reads finished tokens out of state.env rather than patching the parser.

Tech Stack The project is pure Python (99.9% by byte count) targeting Python 3.10+, with typing-extensions as the only runtime dependency for older interpreters. It has no web framework or database dependency — it’s a standalone parsing library. Packaging uses setuptools with pyproject.toml-driven metadata and a dynamic __version__ read from the package; the optional speedup extra swaps in a compiled Cython inline scanner without changing the public API. Documentation is built with Sphinx (shibuya theme, sphinx-design, sphinx-copybutton) and published via a dedicated docs/ tree, and the package exposes a console-script entry point (mistune CLI) defined in pyproject.toml.

Code Quality The project enforces mypy --strict across src/mistune and lints with ruff (import-sort plus pyflakes rules), both wired into CI via .github/workflows/tests.yml. Tests use pytest with pytest-cov and branch coverage tracked through codecov, including full compliance runs against the official CommonMark JSON test-suite fixture (tests/fixtures/commonmark.json) alongside dozens of plugin-specific fixture files (tables, footnotes, math, directives, etc.). filterwarnings = ["error"] in the pytest config turns any stray warning into a test failure, a stricter-than-usual bar for catching deprecation and encoding issues early.

What Makes It Unique Rather than committing to HTML as the only output, Mistune treats the parsed token tree as the stable artifact and renderers as interchangeable consumers — the same parse can emit HTML, a plain-dict AST, Markdown, or reStructuredText through HTMLRenderer, MarkdownRenderer, and RSTRenderer respectively. Its directive framework further generalizes this by supporting two independent directive syntaxes (fenced-code-style via FencedDirective and RST-style via RSTDirective) sharing one DirectiveParser/BaseDirective contract, so custom block syntax (admonitions, includes, table-of-contents markers) can be added without forking the core grammar — a level of syntax extensibility beyond what most CommonMark-focused parsers expose.