python-readability

readability-lxml (python-readability) is a fast Python port of arc90’s Readability algorithm that pulls the primary article content out of noisy, real-world HTML. Given any web page, it strips navigation, sidebars, ads, and boilerplate, then returns a clean summary of the main body text along with the extracted title and author.

Built on lxml for high-speed parsing, it powers content pipelines, web scrapers, RSS readers, and NLP preprocessing where you need just the readable article rather than the full messy DOM. A single Document class exposes everything through a handful of intuitive methods.

What You Get

• A single Document class that wraps raw HTML and exposes summary(), title(), short_title(), and author() methods
• Heuristic content scoring ported from the latest readability.js to reliably isolate the main article
• lxml-powered parsing with automatic character-encoding detection via chardet for robust handling of real-world pages
• HTML cleaning that removes scripts, styles, comments, and nuisance attributes so downstream tools get tidy markup
• Tunable positive/negative keyword hints and image-retention options for fine-grained control over extraction

Common Use Cases

• Extracting clean article bodies for web scrapers and content aggregators
• Preprocessing HTML into readable text for NLP, summarization, and search-indexing pipelines
• Powering read-it-later, RSS, and reader-mode features that strip page chrome
• Building datasets of article text from crawled web pages

Under The Hood

Architecture - The core lives in readability/readability.py, a single ~29KB module whose Document class drives the pipeline: htmls.build_doc parses raw bytes into an lxml tree, candidate <div>/<p> blocks are scored by a set of compiled regexes (positiveRe, negativeRe, unlikelyCandidatesRe) plus link-density and text-length heuristics ported from readability.js, and the highest-scoring container is selected, sanitized, and serialized by summary(). Supporting modules split cleanly: htmls.py handles parsing and title/author extraction, cleaners.py strips nuisance attributes and configures an lxml Cleaner, and encoding.py resolves character sets.

Tech Stack - Pure Python (>=3.8) built on lxml (with the html_clean extra), cssselect for selector support, and chardet for encoding detection, with an optional cchardet speed extra. Packaging is dual-declared via both setup.py and a Poetry pyproject.toml, and it targets Python 3.8 through 3.14.

Code Quality - The tests/ suite contains 16 unittest cases exercising real HTML samples with a timeout decorator guarding against pathological slowdowns. The code favors module-level compiled regexes and small focused functions; error handling centers on an Unparseable exception and lxml fallbacks. It is a mature, stable codebase, though recent development activity is low.

API Design - The public surface is deliberately tiny and ergonomic: construct Document(html) and call summary(), title(), short_title(), or author(). The README’s few-line example gets a user productive immediately, and optional constructor arguments (positive/negative keywords, keep_all_images) expose tuning without complicating the common path.