chardet

chardet is a universal character encoding detector for Python that identifies the encoding of raw byte strings without requiring you to know it in advance. Originally created in 2006 as a port of Mozilla’s universal charset detection library, chardet 7 is a ground-up rewrite that trades the original’s LGPL license for the more permissive 0BSD and delivers dramatically better accuracy and speed while preserving the same public API.

The library runs byte data through a multi-stage detection pipeline — BOM sniffing, magic-number identification, structural probing, byte-validity filtering, and bigram statistical models — to identify the most likely encoding, plus the detected language and a MIME type guess for binary formats. It works both as a one-shot detect() call and as a streaming UniversalDetector for large files, and ships a chardetect CLI for ad hoc use from the shell.

What You Get

• A detect() function that returns encoding, confidence, and language for a byte string in one call
• A detect_all() function that ranks every plausible encoding candidate by confidence
• A UniversalDetector class for incremental/streaming detection of large files without loading them fully into memory
• A chardetect command-line tool for detecting encodings of files directly from the shell
• Encoding-era and include/exclude filters to narrow the candidate set to encodings relevant to your use case

Common Use Cases

• Reading text files of unknown encoding uploaded by users or pulled from third-party sources
• Normalizing scraped HTML/XML content to UTF-8 before parsing
• Processing legacy data exports (CSV, logs, email) written in regional encodings like Windows-1251 or EUC-JP
• Building command-line or batch tooling that needs to sniff encodings of arbitrary files

Under The Hood

Architecture - A detect()/detect_all() call flows into run_pipeline() (pipeline/orchestrator.py), which runs a sequence of discrete stages against the byte data: BOM detection (pipeline/bom.py), a binary-content check (pipeline/binary.py), ASCII detection, magic-number identification (pipeline/magic.py), markup-charset sniffing (pipeline/markup.py), escape-sequence detection for encodings like ISO-2022 (pipeline/escape.py), UTF-8/UTF-16/UTF-32 pattern detection (pipeline/utf8.py, pipeline/utf1632.py), structural scoring for CJK lead-byte diversity and multibyte coverage (pipeline/structural.py), validity filtering (pipeline/validity.py), and finally bigram statistical scoring (pipeline/statistical.py) against per-language BigramProfile models. Language tagging (pipeline/language.py) and postprocessing (pipeline/postprocess.py) run last, producing DetectionResult objects that get converted to dicts and optionally remapped through output_names.py for legacy-compatible naming. The candidate encoding set itself comes from a central EncodingInfo registry (registry.py) that tags each of the 99 supported encodings with metadata used for era/include/exclude filtering. UniversalDetector (detector.py, universaldetector.py) wraps the same pipeline behind an incremental .feed()/.close() interface for streaming use.

Tech Stack - Pure Python, zero runtime dependencies, requires Python 3.10+ and works on PyPy. Packaged with Hatchling plus hatch-vcs for git-tag-derived versioning. An optional hatch-mypyc build hook can compile the hot-path pipeline modules (structural, validity, statistical, utf1632, utf8, escape, language, orchestrator, confusion, magic, ascii, postprocess) to C extensions, which the project credits for a large share of its claimed speedup over prior releases. Dev tooling is Ruff (lint + format, 88-char lines, py310 target) and mypy for type checking, with docs built via Sphinx and hosted on Read the Docs.

Code Quality - 39 test_*.py files under tests/ (plus a scripts/tests suite), run under a strict pytest configuration (--strict-markers, --strict-config) with pytest-cov for coverage and pytest-xdist for parallel execution, plus custom benchmark/serial markers. The codebase uses dataclasses (EncodingInfo, DetectionResult), Literal types for the full encoding registry, and mypy-checked type hints throughout; module docstrings explicitly call out mypyc compilation constraints (e.g. why from __future__ import annotations is intentionally omitted in compiled modules). Input validation (_validate_max_bytes, _validate_encoding) raises on bad arguments rather than silently coercing them.

API Design - The public surface is deliberately small — detect(), detect_all(), UniversalDetector, EncodingEra, LanguageFilter — with every parameter thoroughly docstringed and sensible defaults, so chardet.detect(byte_str) works with zero configuration. Backward compatibility is explicit and intentional: despite being a ground-up rewrite, chardet 7 preserves the exact chardet 5.x/6.x API and default output shape (compat_names, a deprecated should_rename_legacy alias), and the chardetect CLI mirrors the library API almost 1:1 (--minimal, --language, -e/--encoding-era). Documentation is dense: a README quick-start with runnable snippets, a full Sphinx docs site with API reference, ADRs, a how-it-works page, and a FAQ.