Charset Normalizer

Charset Normalizer is a pure-Python library that detects the character encoding of arbitrary byte streams and normalizes their contents to Unicode. Motivated by the limitations of Chardet, it takes a fundamentally different, heuristic approach: instead of relying on trained statistical weights per encoding, it brute-forces decoding across every IANA-supported codec and scores each candidate by how much “mess” and how much linguistic coherence the decoded text exhibits.

Shipped under the permissive MIT license with zero runtime dependencies, it supports 99 encodings, can register custom codecs, detects the spoken language of the text, and guards against UnicodeDecodeError. It is one of the most widely installed packages in the Python ecosystem, serving as the default encoding detector behind the Requests HTTP client.

What You Get

• A high-level Python API (from_path, from_bytes, from_fp) that returns ranked charset matches with a convenient .best() accessor
• A normalizer command-line tool that prints JSON detection results and can rewrite files to Unicode
• A chardet-compatible detect() function for zero-effort migration from Chardet
• Support for 99 IANA encodings, custom codec registration, BOM/signature handling, and spoken-language detection

Common Use Cases

• Decoding user-uploaded files, subtitles, or scraped web content whose encoding is unknown
• Powering the automatic response-encoding detection inside HTTP clients such as Requests
• Batch-normalizing legacy documents from mixed or unknown code pages into UTF-8

Under The Hood

Architecture Detection is driven by from_bytes in src/charset_normalizer/api.py, which iterates over a precomputed multibyte-first ordering of IANA codecs (IANA_SUPPORTED_MB_FIRST) so genuine CJK content short-circuits quickly. For each candidate encoding it decodes the payload in chunks via cut_sequence_chunks (utils.py), scores decoding “mess” with mess_ratio from md.py (a pipeline of MessDetector plugins), and measures linguistic coherence and language with coherence_ratio from cd.py. Signatures and BOMs are stripped through identify_sig_or_bom, and results are ranked and returned as CharsetMatches/CharsetMatch objects (models.py) exposing the chosen encoding, aliases, alternatives, chaos, and coherence.

Tech Stack Written in pure Python targeting 3.7+ with full type hints (py.typed) and from __future__ import annotations throughout. It has zero runtime dependencies. The build uses setuptools with an optional mypyc compilation hook (backend-path = ["_mypyc_hook"]) that compiles hot paths to C extensions for speed while keeping a pure-Python fallback. Large encoding tables live in constant.py.

Code Quality The repository is well tested, with 14 test modules covering base detection, coherence, legacy detect() behavior, edge cases, full detection, binary detection, large payloads, logging, and the CLI. It ships pre-commit hooks, a coverage configuration, an OSV scanner config, and a documented security policy, and carries the OpenSSF best-practices badge. Code is consistently typed and modular.

API Design The public surface is deliberately small and ergonomic: from_path('file'), then str(results.best()) gets you readable text in two lines, while from_bytes/from_fp cover in-memory and stream inputs. A chardet-compatible detect() eases migration, the normalizer CLI emits clean JSON, and extensive Read the Docs documentation covers advanced tuning such as chaos thresholds and codec isolation.