chardetng

chardetng is Mozilla’s next-generation character encoding detector, written in Rust and shipped inside Firefox to guess the encoding of legacy Web pages that don’t declare one. It targets accuracy competitive with ICU and Google’s compact_enc_det (ced) while staying small enough in binary size to be worth bundling alongside encoding_rs, which it depends on for the actual decode step.

Rather than positive pattern matching, chardetng leans on negative matching: encoding errors, disallowed control characters, and implausible byte-pair transitions rule out candidate encodings until a small set of scored guesses remains. It works in a no_std environment without an allocator, and an optional multithreading feature (via rayon and arrayvec) can parallelize per-encoding detection at some binary-size cost.

What You Get

• A EncodingDetector type with feed/guess methods that incrementally scores candidate legacy encodings as bytes arrive
• TLD-aware guessing via tld_may_affect_guess, so a page’s top-level domain can bias detection toward locale-plausible encodings
• Explicit control over whether UTF-8 is considered a valid detection outcome, since browsers should not silently detect UTF-8 without user action
• no_std compatibility, including operation without a heap allocator
• An optional multithreading Cargo feature that parallelizes per-encoding scoring with rayon

Common Use Cases

• Guessing the encoding of HTML/text fetched without a Content-Type charset or a BOM, mirroring what Firefox does for legacy pages
• Building a browser engine, web crawler, or archival tool that must render or index charset-undeclared legacy content correctly
• Adding fallback encoding detection to an encoding_rs-based text pipeline without pulling in a much larger dependency like ICU
• Research or tooling around historical Web content where windows-125x, GBK, Shift_JIS, or other legacy encodings are common

Under The Hood

Architecture: The crate centers on an EncodingDetector struct (src/lib.rs, ~3,900 lines) fed incrementally via feed() and queried via guess(). Detection works by maintaining per-candidate-encoding scores that accumulate bonuses and penalties as bytes stream in — implausible byte-pair transitions, disallowed control characters, and script-mixing patterns subtract score, while frequency-matched character pairs (from src/data.rs, generated from Wikipedia corpora) add it. A separate src/tld.rs module classifies top-level domains so tld_may_affect_guess can bias scoring toward encodings historically associated with a page’s TLD. Because it prioritizes negative matching, a single hard disqualifier (e.g. an invalid byte sequence for an encoding, or a C1 control character) removes that encoding’s candidacy outright rather than merely lowering its score.

Tech Stack: Pure Rust, no_std with no allocator requirement, targeting Rust 1.40+ (no MSRV guarantee across patch releases). It depends on encoding_rs for the Encoding/Decoder types it scores against, memchr for fast byte scanning, and cfg-if for feature-gated code paths. The optional multithreading feature pulls in rayon and arrayvec to evaluate candidate encodings in parallel; detone is a dev-only dependency used in tests.

Code Quality: The crate ships cargo test-driven tests plus a testing-only-no-semver-guarantees-do-not-use feature that exposes internals specifically for the test suite, an unusual but pragmatic pattern for keeping test-only surface out of the public API contract. Constants for scoring bonuses/penalties (e.g. LATIN_ADJACENCY_PENALTY, COPYRIGHT_BONUS) are named and commented with the specific linguistic or browser-compatibility rationale behind each value, which is important given how calibration-heavy this domain is. There is no traditional trait-based abstraction layer — the design favors a dense, heavily-tuned state machine over generic architecture, appropriate for a crate whose entire value is its calibration data.

API Design: The public surface is deliberately small — feed bytes, ask for a guess, optionally hint with a TLD or allow/disallow UTF-8 — which keeps integration low-friction for callers already using encoding_rs. Documentation leans on a companion long-form design write-up (linked from the README) rather than exhaustive rustdoc, so newcomers benefit from reading that essay alongside docs.rs output. The crate explicitly disclaims an accuracy roadmap (“improvements are not planned”), which is honest but means API consumers should not expect the detection heuristics themselves to evolve much beyond bug fixes.