sphinxcontrib-serializinghtml

sphinxcontrib-serializinghtml is an official Sphinx extension that adds two alternative output builders — json and pickle — to the Sphinx documentation generator. Instead of rendering fully-formed HTML pages, these builders dump each page’s rendered context (title, body, table of contents, sidebar data) as a serialized JSON or pickle file per document, plus a shared global context file and a search index.

This serialized output is not meant to be browsed directly. It exists so a separate web application — historically Sphinx’s own sphinx.websupport package and similar downstream tools — can load page data programmatically, merge it into a custom template layer, and serve documentation dynamically rather than as static files. The extension was originally part of Sphinx core and was split out into this standalone package to keep Sphinx’s own dependency surface smaller.

The package is intentionally minimal: two builder classes (JSONHTMLBuilder, PickleHTMLBuilder) that subclass Sphinx’s StandaloneHTMLBuilder and override how page context gets written to disk, plus a thin jsonimpl module that wraps the standard library json module with a custom encoder for Sphinx’s translation-proxy strings.

What You Get

• A json Sphinx builder that writes each page’s context as a .fjson file plus a shared globalcontext.json and searchindex.json
• A pickle Sphinx builder that writes the same data using Python’s pickle module for faster, binary-native loading in Python consumers
• A custom SphinxJSONEncoder that correctly serializes Sphinx’s lazy translation-proxy strings (UserString instances) to plain JSON strings
• Automatic copying of the environment pickle and a last_build touch-file so a consuming web app can detect when it needs to reload its cache
• Zero required runtime dependency on Sphinx itself in the package metadata, avoiding a circular dependency between Sphinx and its own extension

Common Use Cases

• Building a custom documentation web application that reads page data as JSON rather than scraping rendered HTML
• Feeding Sphinx-generated documentation into a search or indexing pipeline that expects structured JSON records
• Powering Sphinx’s own websupport integration for dynamic, database-backed documentation sites
• Pre-processing documentation content for translation or content-migration tooling that needs structured page context rather than markup

Under The Hood

Architecture The package defines a single abstract base, SerializingHTMLBuilder, which subclasses Sphinx’s StandaloneHTMLBuilder (in sphinxcontrib/serializinghtml/__init__.py) and overrides get_target_uri, handle_page, and handle_finish to intercept the point where Sphinx would normally render a Jinja template to HTML. Instead, dump_context() serializes the page’s template context dict directly via a pluggable implementation attribute (a module exposing dump/dumps/load/loads, mirroring the stdlib json/pickle interface). JSONHTMLBuilder and PickleHTMLBuilder are thin concrete subclasses that just set implementation, output suffixes, and filenames. handle_finish() also copies the environment pickle and search index alongside a last_build sentinel file so an external consumer can detect staleness. The design cleanly reuses Sphinx’s page-rendering pipeline up to the templating boundary, swapping only the final output step — a small, surgical override rather than a parallel implementation. Tech Stack Pure Python (3.9+), built with the flit_core backend declared in pyproject.toml. Its only structural dependency is Sphinx itself (sphinx.application, sphinx.builders.html, sphinx.locale, sphinx.util.osutil), which is intentionally left out of the package’s declared runtime dependencies to avoid a circular install requirement — Sphinx installs this extension, not the other way around. The jsonimpl module wraps the standard library’s json module; the pickle builder uses the standard library pickle module directly with no wrapper. Development tooling is Ruff for linting (configured in .ruff.toml with a fairly broad rule set including bugbear, comprehensions, and datetime-aware checks) and MyPy in strict mode for type checking. Code Quality The test suite (tests/test_serializinghtml.py plus a tests/roots/test-basic fixture project and conftest.py) uses pytest and Sphinx’s own test harness to build a minimal fixture documentation tree with each builder and assert on the resulting output files — a small but targeted test surface appropriate for the package’s narrow scope. Type hints are used throughout the source with from __future__ import annotations, and MyPy strict-mode settings (disallow_untyped_defs, disallow_any_generics, warn_unused_ignores, etc.) are enforced in pyproject.toml. CI runs via GitHub Actions (test.yml) on each push/PR. Naming and structure closely mirror the equivalent code that used to live inside Sphinx core, so conventions are consistent with the parent project. What Makes It Unique The package’s only real distinguishing idea is the pluggable implementation protocol — a builder subclass just points implementation at any object exposing dump/dumps/load/loads, which is how the same base class supports both JSON and pickle output with almost no duplicated logic. There’s nothing novel about the serialization itself; this is a narrowly-scoped, historically-motivated extraction from Sphinx core rather than a general-purpose tool, and it functions correctly as exactly that.