ReadabiliPy

ReadabiliPy extracts the readable content from a raw HTML page and returns it as a structured Python dictionary — title, byline, publish date, cleaned HTML content, and a list of plain-text paragraphs — instead of leaving callers to parse boilerplate-laden markup themselves. It was built at the Alan Turing Institute to give research and content pipelines a consistent, testable way to pull article text out of arbitrary scraped HTML.

The library can operate in two modes: by default it wraps Mozilla’s own Readability.js (the engine behind Firefox’s Reader View) via a Node.js subprocess, or it can run a pure-Python HTML5lib/BeautifulSoup-based simplifier when Node.js isn’t available, so the same API keeps working in restricted or Node-less deployment environments. It ships both as an importable library (simple_json_from_html_string) and as a readabilipy command-line tool for converting saved HTML files directly to JSON.

What You Get

• A single function, simple_json_from_html_string, that returns a dictionary with title, byline, ISO-formatted date, cleaned HTML content, simplified HTML, and plain-text paragraphs
• A readabilipy CLI that reads HTML from a file or stdin and writes extracted article JSON to a file or stdout
• Two interchangeable extraction backends: Mozilla’s Readability.js (via an on-demand npm-installed Node.js subprocess) or a pure-Python html5lib/BeautifulSoup simplifier
• Optional SHA-256 content-digest and hierarchical node-index attributes on the simplified HTML for diffing or addressing specific paragraphs
• A scored, prioritized date-extraction pass that mines meta tags and microdata and normalizes the result to ISO 8601

Common Use Cases

• Building read-it-later or reading-list apps that need clean article text stripped of ads and navigation chrome
• Preparing scraped web pages for LLM/RAG ingestion using the plain-text paragraph output
• Research pipelines that need consistent, structured extraction of article content and metadata across large sets of news pages
• Deployments without a Node.js runtime, where the pure-Python fallback keeps extraction working without losing functionality

Under The Hood

Architecture Everything funnels through simple_json_from_html_string in readabilipy/simple_json.py, which branches on a use_readability flag: when true and have_node() confirms a usable Node.js runtime with an installed javascript/node_modules directory (auto-triggering run_npm_install from utils.py on first use), it shells out via subprocess.run to javascript/ExtractArticle.js, passing the input HTML through a temp file and reading Mozilla’s Readability.js output back as JSON; when false, or Node is unavailable, it falls back to simple_tree.py’s simple_tree_from_html_string, a BeautifulSoup/html5lib pipeline that runs through remove_metadata → strip_attributes → remove_blacklist → unwrap_elements → process_special_elements/process_unknown_elements → consolidate_text → unnest_paragraphs → insert_paragraph_breaks → wrap_bare_text → normalise_strings → recursively_prune_elements, all defined in simplifiers/html.py. Both paths converge on a shared post-processing stage (plain_content/plain_element) that recursively flattens the resulting tree into leaf-level plain text with optional SHA-256 content digests and hierarchical node-index attributes, while extractors/extract_date.py and extractors/extract_title.py independently mine metadata via a scored XPath list. It’s a thin orchestration layer with a real subprocess boundary at its core — if the Readability.js bridge broke, only the pure-Python simplifier path would keep working.

Tech Stack A Python package (python_requires>=3.6) depending on beautifulsoup4, html5lib, lxml, and regex for the pure-Python path; the Readability.js path additionally requires an external Node.js runtime (v10+) and its own javascript/package.json wrapping Mozilla’s @mozilla/readability, installed on demand via a shelled-out npm install. Distribution is a standard setup.py/PyPI package with a single console-script entry point (readabilipy), no async runtime, database, or web framework involved. CI runs via GitHub Actions (lint.yml, test.yml) executing pytest alongside pylint/pycodestyle/pyflakes, with coverage reported to Coveralls.

Code Quality Tests live under tests/ (test_article_extraction.py, test_benchmarking.py, test_date_functions.py, test_extract_element.py, test_html_elements.py, test_javascript.py) using pytest with a shared checks.py helper that diffs extraction output against golden fixture JSON files built from real scraped articles, plus pytest-benchmark timing tests and a strict .pylintrc/pycodestyle/pyflakes lint pass wired into CI. Error handling favors graceful degradation over exceptions — have_node() and run_npm_install() print warnings and fall back silently when Node/npm are missing, while genuine subprocess failures in simple_json.py propagate via CalledProcessError after logging stderr. There are no type annotations or static type checking anywhere in the codebase; naming is consistent snake_case with short, single-purpose functions.

API Design The public surface is minimal and predictable: one function call, simple_json_from_html_string(html, content_digests=False, node_indexes=False, use_readability=False), always returns a dict with the same fixed key set (missing fields default to None rather than being absent), so callers never need to schema-sniff the response. The CLI mirrors the library flags one-to-one (-c/-n/-p), keeping behavior consistent between the two entry points, and even reports whether the Node.js backend is actually available via --version. Documentation is a single well-organized README covering installation, CLI usage, and every returned field’s semantics, but there’s no separate docs site, API reference, or type stubs, so discoverability relies on reading source or README rather than IDE autocompletion.