unicode-normalization

unicode-normalization is a Rust crate that implements Unicode character composition and decomposition exactly as specified by Unicode Standard Annex #15. It gives Rust programs iterator-based access to the four canonical normalization forms (NFC, NFD, NFKC, NFKD) plus quick-check and stream-safe variants, using generated lookup tables built directly from the Unicode Character Database.

As a foundational text-processing primitive it is depended on transitively by huge swaths of the Rust ecosystem (URL parsing, IDNA/domain handling, text search, and Unicode-aware string libraries), making correctness and no_std portability its primary design goals rather than a large feature surface.

What You Get

• Iterator adaptors for NFC, NFD, NFKC, and NFKD normalization directly on string/char types via the UnicodeNormalization trait
• Quick-check functions (is_nfc, is_nfd, is_nfkc, is_nfkd, and stream-safe variants) to test normalization status without full transformation
• Stream-safe text processing (StreamSafe) to bound combining-character run lengths per UAX #15 §13
• no_std + alloc compatibility for embedded and constrained environments (default-features = false)
• Character-level composition/decomposition primitives in the char submodule for building custom normalization pipelines

Common Use Cases

• Normalizing user input (search queries, form fields, filenames) so visually-identical strings compare equal
• IDNA/domain-name and URL normalization as a dependency of crates like url and idna
• Preprocessing text before hashing, indexing, or full-text search to avoid Unicode-equivalence duplicates
• Validating that stored or transmitted strings are already in a canonical form before further processing

Under The Hood

Architecture — The crate is organized around a small set of focused modules: decompose.rs and recompose.rs implement the core Decompositions/Recompositions iterator adaptors that wrap a Chars iterator, quick_check.rs implements the UAX #15 quick-check algorithm for constant-time-ish normalization-form testing, stream_safe.rs implements the stream-safe text process, and tables.rs (22.8k generated lines) holds the perfect-hash-indexed Unicode Character Database tables (perfect_hash.rs implements the lookup). lib.rs re-exports the public surface as extension traits on char/str via UnicodeNormalization, keeping the API surface thin relative to the data-heavy implementation.

Tech Stack — Pure Rust, edition 2018, MSRV 1.36, with a single runtime dependency (tinyvec, used with the alloc feature for small-buffer decomposition results without heap allocation in the common case). The crate is #![cfg_attr(not(feature = "std"), no_std)] and gated #![deny(missing_docs, unsafe_code)], so it ships with zero unsafe code and mandatory doc coverage. Generated tables are produced by scripts in scripts/ against the Unicode Character Database rather than hand-maintained.

Code Quality — Tests live in tests/ (property-style tests against tests/data, CJK compat-variant tests, and a stream-safe regression test) plus inline #[cfg(test)] mod test in src/. The deny(missing_docs) lint enforces documented public API, and deny(unsafe_code) rules out an entire class of memory-safety bugs given how deeply this crate sits in the dependency graph. Benchmarks exist under benches/, and a fuzz/ directory indicates fuzz-testing of the normalization routines.

API Design — The extension-trait pattern ("...".nfc().collect::<String>()) is idiomatic Rust and requires zero setup beyond a single use statement; quick-check functions mirror the transform functions 1:1 (is_nfc/nfc), which keeps the mental model simple. The tradeoff is a fairly low-level API — callers get normalization primitives, not higher-level text-processing helpers — appropriate for a crate meant to be a dependency of other libraries rather than an end-user tool.