webencodings

webencodings is a small, dependency-free Python library that implements the WHATWG Encoding standard’s label-to-codec resolution rules — the same rules browsers use to make sense of Content-Type: text/html; charset=... declarations, HTML <meta charset> tags, and byte-order marks on the legacy web. Rather than reimplementing character encoding and decoding itself, it maps the dozens of encoding aliases and browser-specific quirks (like US-ASCII and iso-8859-1 both meaning windows-1252 on the web) onto Python’s own codecs module, and adds BOM detection plus two encodings the stdlib lacks (x-user-defined and replacement).

It’s most often found as a quiet dependency inside HTML/CSS parsing libraries — html5lib, tinycss2, and cssselect2 all use it — wherever code needs to correctly guess the encoding of untrusted, possibly mislabeled web content instead of assuming UTF-8.

What You Get

• Label-to-encoding resolution matching the WHATWG Encoding spec’s get an encoding algorithm, covering every encoding label defined by the standard.
• Automatic BOM (byte-order-mark) detection for UTF-8, UTF-16LE, and UTF-16BE that takes precedence over any declared fallback encoding.
• One-shot decode()/encode() functions plus streaming IncrementalDecoder/IncrementalEncoder and pull-based iter_decode()/iter_encode() generators for chunked input.
• Two encodings the Python stdlib doesn’t ship — x-user-defined and replacement — implemented via codecs.charmap_encode/charmap_decode.

Common Use Cases

• Resolving the encoding declared in an HTML document’s <meta charset> tag or HTTP Content-Type header before parsing.
• Normalizing legacy/browser-only encoding aliases (latin1, US-ASCII, windows-874, shift_jis) to the correct Python codec name.
• Detecting and stripping a UTF-8/UTF-16 byte-order mark before decoding a byte stream.
• Streaming-decoding a response body chunk-by-chunk as it arrives over the network, without buffering the whole payload first.

Under The Hood

Architecture The library is deliberately flat: webencodings/__init__.py exposes the entire public API (lookup, decode, encode, iter_decode, iter_encode, IncrementalDecoder, IncrementalEncoder), webencodings/labels.py holds a generated LABELS dict mapping every WHATWG label string to its canonical encoding name (produced by mklabels.py from the spec), and webencodings/custom.py implements the two encodings the stdlib lacks by subclassing codecs.Codec/IncrementalEncoder/IncrementalDecoder/StreamWriter/StreamReader. An Encoding class wraps a canonical name with a stdlib CodecInfo object and is cached module-wide in a CACHE dict to avoid repeated codecs.lookup() calls. Every public entry point funnels through a shared _get_encoding() coercion step, so the Encoding/CACHE design is the one abstraction the rest of the library depends on.

Tech Stack Pure Python 3.10+ with zero runtime dependencies — only the stdlib codecs module is used. Packaging uses flit_core via pyproject.toml (PEP 517), and releases are published to PyPI through a GitHub Actions workflow using OIDC trusted publishing (id-token: write with pypa/gh-action-pypi-publish). Documentation is built with Sphinx (furo theme) and hosted separately; the test extra installs pytest and ruff for linting.

Code Quality A single flat pytest-style test file exhaustively exercises the public API: label lookup and ASCII case-insensitivity, the legacy remapping table, every WHATWG label round-tripped through decode/encode/iter_decode/iter_encode/the incremental classes, invalid-label LookupErrors, BOM detection across all three supported encodings, legacy CJK codec remapping, and both custom codecs. There are no type hints or mypy step in CI; ruff is configured with a defined rule set and enforced single-quote style instead. Error handling is intentionally thin — it raises stdlib LookupError for unknown labels and otherwise defers to Python’s own codec error-handler machinery rather than reimplementing it.

API Design The public surface is five functions and two streaming classes, all accepting either a label string or an already-resolved Encoding object interchangeably. Sensible defaults follow platform convention (decode() defaults to errors='replace' like browsers, encode() defaults to errors='strict' like Python). Every public symbol carries a complete Sphinx-style docstring, including a worked doctest on ascii_lower(), and a dedicated docs site adds first-steps, use-case, and API reference pages beyond what the source alone provides — unusually thorough documentation for a library this small.