html5lib

html5lib is a Python implementation of the WHATWG HTML5 parsing algorithm, built to match how real browsers parse HTML byte-for-byte, including malformed and “tag soup” markup that a strict XML-style parser would reject. It exposes pluggable tree builders so parsed documents can come out as xml.etree trees, lxml trees, or a generic DOM-like structure, plus tree walkers and a serializer for converting parsed trees back into HTML or XHTML output.

Because it targets the actual HTML5 parsing spec rather than a simplified approximation, html5lib is the reference many other Python HTML tools (including BeautifulSoup, which supports it as a parser backend) rely on when spec-accurate, browser-matching parsing behavior matters more than raw speed.

What You Get

• A tokenizer and tree-construction parser that follows the WHATWG HTML5 parsing spec, including its error-recovery rules for malformed markup
• Pluggable tree builders producing xml.etree.ElementTree, lxml.etree, or a built-in generic tree representation
• Tree walkers that traverse any of the supported tree formats uniformly for downstream processing
• A serializer for converting parsed trees back into HTML or XHTML strings with configurable formatting options
• An HTML sanitizer filter (html5lib.filters.sanitizer) for stripping unsafe tags/attributes from untrusted markup
• A large conformance test suite validated against the shared html5lib-tests corpus used across HTML5 parser implementations in multiple languages

Common Use Cases

• Parsing real-world, imperfect HTML (scraped pages, user-submitted content, legacy documents) the same way a browser would, rather than failing on malformed markup
• Serving as the underlying parser for higher-level scraping/processing libraries like BeautifulSoup when spec accuracy matters more than raw parsing speed
• Sanitizing untrusted HTML input before storage or display, using the built-in sanitizer filter

Under The Hood

Architecture: The parser is split into a tokenizer (_tokenizer.py) that turns raw bytes/text into HTML tokens per the WHATWG state-machine spec, and html5parser.py which implements the tree-construction algorithm’s insertion modes on top of those tokens. Output isn’t tied to one tree representation — treebuilders/ provides adapters for etree, lxml, and a built-in DOM-like tree, treewalkers/ provides a common traversal interface over any of them, and filters/ (including the sanitizer) can be chained onto a tree walker before serialization. _inputstream.py handles the encoding-detection algorithm (BOM sniffing, meta charset detection, chardet fallback) that the HTML5 spec requires for correct parsing of documents without explicit encoding declarations.

Tech Stack: Pure Python (with an optional lxml tree-builder dependency and chardet/charset_normalizer for encoding detection), supporting both Python 2 and 3 codepaths historically, tested via pytest and tox across Python version matrices with an oldest-supported-dependency lockfile (requirements-oldest.txt).

Code Quality: The test suite pulls from the shared html5lib-tests conformance corpus (tokenizer tests, tree-construction tests, serializer tests, encoding tests) that’s also used by HTML5 parser implementations in other languages, giving strong confidence that parsing behavior actually matches the spec rather than just internal expectations. Flake8 linting is wired via a dedicated script. Recent commit activity has slowed (health score flags low recent activity and infrequent maintenance), consistent with a mature, spec-complete parser that needs fewer changes once conformant.

API Design: The top-level html5lib.parse() function covers the common case with a single call, while HTMLParser, tree-builder selection (treebuilder='lxml'), and the tree-walker/filter/serializer chain are available for more advanced pipelines. This layered design — simple default, composable advanced path — keeps casual users unblocked while still exposing the pieces (tokenizer, tree builder, sanitizer) that library authors building on top of html5lib (like BeautifulSoup) need to integrate against.