encoding_rs

encoding_rs is a Rust implementation of the WHATWG Encoding Standard, the specification that defines every character encoding a web browser is required to understand. Originally built for and shipped inside Firefox since Gecko 56, it converts bytes in legacy encodings such as Shift_JIS, GBK, Big5, EUC-JP/KR, and the ISO-8859 family to and from valid UTF-8 or UTF-16, matching exactly how a browser resolves a page’s character set from its <meta charset>, an HTTP header, or a BOM.

Because it grew out of a browser engine’s requirements, the crate favors streaming, allocation-light APIs that work equally well decoding a single in-memory buffer or a network response arriving in unpredictable chunks, and its mem module adds fast routines for classifying and converting in-RAM text (ASCII/Latin1/UTF-16 checks and conversions) that Gecko needs outside of any I/O boundary.

What You Get

• Full WHATWG Encoding Standard coverage - every encoding a browser must support (UTF-8, UTF-16LE/BE, Shift_JIS, EUC-JP/KR, Big5, GBK/GB18030, ISO-2022-JP, the ISO-8859 family, windows-125x, and more) behind a single Encoding type.
• Streaming and non-streaming decode/encode APIs - convert a whole buffer at once or feed a Decoder/Encoder chunks as they arrive from the network, with borrow-avoiding fast paths for the common UTF-8 case.
• Label resolution matching browser behavior - resolve textual labels (from <meta charset>, HTTP Content-Type, or a BOM) to the correct Encoding the same way document.characterSet would.
• A mem module for in-RAM text - ASCII/Latin1/UTF-16 validity and range checks, right-to-left hint detection, and conversions between UTF-8, UTF-16, and Latin1 without going through an I/O boundary.
• no_std support - works without the standard library, and without an allocator at all if the default alloc feature is disabled.

Common Use Cases

• Decoding HTTP responses or files whose encoding is only known from a textual label, the same way a browser resolves <meta charset>.
• Building a text editor, mail client, or archive tool that must round-trip legacy Windows/Mac/CJK encodings to and from Unicode.
• Implementing an FFI-friendly decode/encode boundary for a C or C++ host application via the sibling encoding_c bindings.
• Classifying or converting large in-memory text buffers (ASCII/Latin1/UTF-16 checks) in a performance-sensitive Rust or embedded (no_std) context.

Under The Hood

Architecture: encoding_rs organizes each encoding family into its own module (big5.rs, euc_jp.rs, euc_kr.rs, gb18030.rs, iso_2022_jp.rs, shift_jis.rs, single_byte.rs, utf_8.rs, utf_16.rs, x_user_defined.rs, replacement.rs), each exposing a private decoder/encoder implementation. The generated variant.rs (explicitly marked “PLEASE DO NOT EDIT”) wraps them into two closed enums, VariantDecoder/VariantEncoder, whose match arms form the explicit dispatch table that lets the public Decoder/Encoder structs in lib.rs stay Sized and stack-allocatable instead of relying on trait objects. handles.rs defines lifetime-carrying Handle wrapper structs that couple bounds-checking to slice reads/writes, and macros.rs generates the shared streaming decode/encode loop bodies used across all the per-encoding modules so the hot loop is written once and monomorphized per encoding. data.rs (2.5MB, generated by generate-encoding-data.py from WHATWG’s encodings.json) holds the character-mapping tables the per-encoding modules index into. The public surface centers on a single &‘static Encoding type (one static instance per encoding, e.g. SHIFT_JIS) that hands out Decoder/Encoder values and label resolution, plus the mem module for in-RAM text operations.

Tech Stack: Pure Rust, edition 2024, minimum rust-version 1.88. Runtime dependencies are minimal: cfg-if unconditionally, plus three optional ones — serde (behind the serde feature, for serializing the &‘static Encoding reference) and any_all_workaround/rustversion (pulled in only by the opt-in simd-accel feature, which uses core::simd and is nightly/version-sensitive by the crate’s own documentation). Dev-dependencies (serde_derive, bincode, serde_json) are test-only. The crate supports no_std (the default feature is just alloc; disabling it drops the Vec/String/Cow-returning API surface). The character tables in src/data.rs are generated ahead of time by a checked-in Python script rather than a build.rs, so consumers never need Python to build the crate, and [profile.release] sets lto = true.

Code Quality: Tests are extensive and split across two layers: dense inline #[test] modules inside nearly every source file (mem.rs alone has 41, lib.rs 46, utf_16.rs 12, single_byte.rs and utf_8.rs 8 each), giving per-encoding round-trip and edge-case coverage, plus a top-level tests/ directory (streaming_empty_chunks.rs, test_data.rs) exercising streaming behavior against WHATWG’s own test-data corpus, and a fuzz/ directory with dedicated fuzz targets. Errors are represented as explicit DecoderResult/EncoderResult enums returned from every conversion call rather than panics, matching the browser/FFI-facing design goal. Naming is consistent and spec-literal (e.g. SHIFT_JIS, EUC_JP, ISO_8859_2), generated files are clearly marked as such and kept separate from hand-written code, and the crate-root clippy allows are itemized rather than a blanket suppression.

API Design: The entire public API funnels through one type, &‘static Encoding, obtained either as a named constant (SHIFT_JIS) or via label resolution (Encoding::for_label), which then hands out streaming Decoder/Encoder values or exposes one-shot decode/encode convenience methods returning (Cow<str>, &Encoding, bool) — a small, learnable surface for a domain of 40+ legacy encodings that could easily have sprawled into dozens of per-encoding types. The one documented API surprise (encoders always target an encoding’s Web output encoding, so UTF_16LE/UTF_16BE/replacement can’t be used as encode targets) is called out explicitly in the crate’s own docs. The streaming API requires tracking buffer state across chunk boundaries, which is inherently more boilerplate-heavy than a one-shot conversion, but this is an unavoidable consequence of the browser-streaming use case rather than incidental complexity, and the crate offers simpler non-streaming decode/encode methods for the common in-memory case. Doc coverage is strong: a large module-level doc with runnable examples for both streaming and non-streaming decode, plus an external long-form design write-up linked from the README.