decancer

decancer is a Rust library for detecting and normalizing disguised or obfuscated text, commonly used to bypass content moderation, search matching, and username filters. It strips diacritics, homoglyphs, leetspeak substitutions, and other unicode confusables — filtering roughly 222,000 codepoints by default — while remaining aware of Unicode’s bidirectional text algorithm so right-to-left scripts are handled correctly rather than naively reordered.

Under the hood it forgoes hash-map lookups in favor of binary search over precompiled, densely packed codepoint tables, keeping cure operations fast even on longer strings. The core crate exposes a small API — cure() and cure_char() — with an Options builder for toggling capitalization retention, ASCII/alphanumeric-only output, leetspeak handling, and RTL treatment, and it powers official bindings for Node.js/WASM, Java, Go, and native C/C++.

What You Get

• Binary-search codepoint matching against roughly 222,557 precompiled confusable mappings
• A CuredString wrapper type with contains, censor, and censor_multiple helpers for post-processing
• Unicode bidi-aware string reordering so right-to-left text is cured the way it’s actually displayed
• An Options builder for retaining capitalization, restricting to ASCII/alphanumeric output, and toggling leetspeak translation
• Official language bindings for Node.js/WASM, Java, Go, and native C/C++ built on the same Rust core

Common Use Cases

• Chat and community moderation systems flagging usernames or messages that use lookalike characters to evade filters
• Search and autocomplete normalization so stylized unicode text matches its plain-ASCII query
• Profanity and slur filters that need to see through leetspeak and homoglyph substitutions before matching
• Game server anti-evasion tooling that normalizes player names and chat before running them through blocklists

Under The Hood

Architecture The crate splits responsibilities across lib.rs (public API + orchestration), bidi/ (mod.rs, class.rs, level.rs, paragraph.rs, brackets.rs implementing the Unicode bidirectional algorithm), codepoints.rs (binary-search lookups over a packed binary table embedded via include_bytes!), options.rs (a bitflag-style Options builder), similar.rs (near-match logic), string.rs (the CuredString wrapper), translation.rs (a Translation enum for single- vs multi-character outputs), leetspeak.rs (a regex-based pass gated behind a feature flag), and util.rs. cure() runs a two-phase pipeline: first_cure_pass classifies each codepoint’s bidi Class and groups it into paragraphs, then cure_reordered computes explicit and isolating-run-sequence levels per the Unicode Bidi Algorithm before visually reordering runs and feeding each character through cure_char_inner. Feature flags (options, separators, leetspeak, serde) are compiled out via #[cfg], so callers using only the cure! macro pay no runtime cost for options they’ve disabled.

Tech Stack Rust 2024 edition, with rust-version pinned to 1.87.0. Runtime dependencies are minimal and optional: serde (behind a serde feature, for Options (de)serialization) and regex (behind the leetspeak feature only). Dev-dependencies include proptest for property-based testing, criterion for benchmarking, rand, and censor for comparison in tests. Precompiled binary lookup tables (bin/codepoints.bin, bin/bidi.bin) are embedded at compile time rather than parsed at runtime. The repo lays core/ (the pure-Rust crate) alongside bindings/{node,wasm,java,go,native}, each wrapping the core crate for its target ecosystem via napi/wasm-bindgen, JNI, cgo, and a C header respectively. CI runs via .github/workflows/CI.yml.

Code Quality Tests live inline (#[cfg(test)] mod tests, retain_tests.rs) alongside a dedicated tests.rs with extensive hand-written cases covering bidi edge cases, option combinations, and confusable mappings, plus proptest-driven property tests. Clippy’s all, pedantic, and nursery lint groups are all set to warn at the crate level, unsafe_code is forbidden via a crate-level lint, and broken_intra_doc_links is denied. Error handling favors explicit Result<_, Error> over panics for bidi malformation cases, and rustdoc comments include runnable doctest examples embedded straight from the README.

What Makes It Unique decancer’s differentiator over similar unicode-cleaning crates is that it runs Unicode’s bidirectional algorithm before curing, so mixed-direction and right-to-left strings are processed in the order they’d actually be rendered rather than naive character order — most confusable-stripping libraries skip this and mishandle RTL scripts. It also ships first-class multi-language bindings (Node/WASM, Java, Go, native C) generated from the same core crate and binary lookup tables, and leans on Cargo feature flags to eliminate the customization machinery entirely from the compiled binary when unused.