lxml

lxml is a Python binding for the libxml2 and libxslt C libraries, giving Python code access to industrial-strength XML and HTML parsing, validation, XPath, and XSLT without sacrificing speed. Its lxml.etree module mirrors the standard library’s ElementTree API, so existing code can adopt it with minimal changes while gaining access to full XPath 1.0 expressions, XSLT transformations, RelaxNG and XML Schema validation, and C14N canonicalization.

Beyond the ElementTree-compatible core, lxml ships lxml.html for lenient real-world HTML parsing and cleanup, lxml.objectify for attribute-style data-binding access to XML trees, and a soup-parser mode for recovering structure from badly formed markup. With over 250 million monthly downloads from PyPI, it is a de facto standard wherever Python needs to parse, generate, or transform XML and HTML at scale.

What You Get

• An lxml.etree API that is a drop-in superset of the standard library’s xml.etree.ElementTree, easing migration
• Full XPath 1.0 and XSLT 1.0 support for querying and transforming documents
• RelaxNG, XML Schema (XSD), and DTD validation built on libxml2’s native validators
• lxml.html for parsing and cleaning real-world, often malformed HTML documents
• lxml.objectify for attribute-style, data-binding access to XML trees
• C14N canonicalization and incremental/iterative parsing (iterparse) for large documents

Common Use Cases

• Scraping and cleaning HTML from live websites where markup is malformed or inconsistent
• Parsing and validating large XML feeds or documents (e.g. SOAP, RSS, sitemaps) against XSD/RelaxNG schemas
• Running XSLT transformations to convert XML into HTML, other XML dialects, or plain text
• Extracting data from documents with XPath expressions instead of manual tree traversal
• High-throughput XML processing where the pure-Python xml.etree.ElementTree is too slow

Under The Hood

Architecture lxml is a thin, tightly-integrated Cython layer over the libxml2 and libxslt C libraries rather than a pure-Python tree implementation. The compiled core lives in src/lxml/etree.pyx, a single large extension module assembled by #include-ing dozens of .pxi fragments (parser.pxi, xslt.pxi, xmlschema.pxi, xpath.pxi, proxy.pxi, apihelpers.pxi, and others) so the whole surface compiles into one shared object for performance. The includes/ directory holds .pxd files that declare the libxml2/libxslt C API to Cython, and proxy.pxi handles the Python/C object proxying that lets Python-level Element objects wrap underlying libxml2 tree nodes safely. Higher-level, pure-Python modules — objectify.pyx, sax.py, builder.py, cssselect.py, and the html/ subpackage — sit on top of this compiled core and delegate the actual parsing, validation, and transformation work down to libxml2/libxslt via direct C calls. Because so much is compiled into one module by design, changes to shared internals like apihelpers.pxi or proxy.pxi ripple across the whole library, but the API surface itself is cleanly partitioned by concern (etree, html, objectify, sax).

Tech Stack The project targets Python 3.9+ and is built with Cython 3.3+ against statically or dynamically linked libxml2 and libxslt (pinned versions such as libxml2 2.14.6 and libxslt 1.1.43 for released wheels, configured via cibuildwheel). The build itself is a custom setup.py/setupinfo.py/versioninfo.py pipeline rather than a declarative pyproject.toml metadata block, reflecting the complexity of compiling against a C toolchain across many platforms and architectures (x86_64, aarch64, i686, ppc64le, armv7l, riscv64, plus Windows and macOS universal2 builds). Optional extras integrate cssselect, html5lib, and BeautifulSoup4 for CSS-selector queries and HTML5-compliant or soup-style parsing. CI and release wheels are produced through GitHub Actions and cibuildwheel.

Code Quality The src/lxml/tests directory contains a large, long-standing unittest-based suite (test_etree.py, test_html5parser.py, test_elementtree.py, test_dtd.py, test_xpathevaluator.py, and dozens more) alongside doctest-driven tests exercised via doc/*.txt files and a doctestcompare.py helper, plus fuzz-testing entry points (fuzz_xml_parse.py). With over 7,100 commits and 44 tracked releases, the project shows a long, consistently maintained history. Type safety is enforced at the Cython level through .pxd declarations rather than Python type hints, and correctness of the C-library bindings is validated continuously through the extensive test matrix run across build targets in CI.

API Design lxml’s central design decision is API compatibility: lxml.etree mirrors the standard library’s ElementTree interface closely enough that many programs can switch by changing an import, then opt into XPath, XSLT, and schema validation as needed rather than reaching for separate libraries. This consolidates functionality — parsing, querying, transforming, validating, and HTML cleanup — that in other language ecosystems is typically split across several packages. The tradeoff is a real learning curve for the C-library-backed pieces of the API (memory management semantics, thread-safety considerations, the objectify data-binding style) that go beyond basic tree manipulation, though the extensive doc/ directory (FAQ, performance notes, parsing guide, XPath/XSLT guide) and README are aimed squarely at bridging that gap.