unicode-segmentation

unicode-segmentation is a Rust crate maintained by the unicode-rs organization that implements the boundary-finding rules from Unicode Standard Annex #29. It provides iterators that split &str values into extended grapheme clusters, words, and sentences, handling the subtleties of combining marks, emoji modifier sequences, and locale-independent word/sentence punctuation correctly instead of naively iterating over chars.

Because text processing that only iterates over Rust chars (Unicode scalar values) frequently misrepresents what a human perceives as a single character, this crate is a foundational dependency across the Rust ecosystem — used by text editors, terminal emulators, formatters, and CLI tools that need accurate cursor movement, line wrapping, or word counting on arbitrary Unicode text.

What You Get

• A graphemes(is_extended: bool) iterator that yields extended grapheme clusters (what users perceive as a single visual character, including combining marks and emoji ZWJ sequences)
• unicode_words() and split_word_bounds() iterators for locale-independent word segmentation, including handling of apostrophes, numbers, and punctuation
• unicode_sentences() for splitting text into sentence-like units per the Unicode sentence-boundary rules
• A GraphemeCursor type supporting random-access and bidirectional boundary lookup for editors that need to move a cursor by one grapheme at a time
• no_std compatibility (no libstd dependency) so it can be used in embedded and freestanding Rust contexts
• Precompiled Unicode property tables (tables.rs) regenerated per Unicode release so behavior stays current with the latest Unicode Character Database

Common Use Cases

• Text editors and terminal emulators computing correct cursor movement and line-wrapping across multi-codepoint emoji or accented characters
• CLI tools and formatters that need accurate word or character counts for user-facing text (e.g. line-length limits, truncation with ellipsis)
• String-processing libraries and parsers (e.g. syntax highlighters, diff tools) that need to iterate text in units a human would recognize as one character
• Internationalized applications performing sentence or word splitting without pulling in a full ICU dependency

Under The Hood

Architecture: The crate is organized into four focused modules — grapheme.rs (extended grapheme cluster iteration and the GraphemeCursor random-access type), word.rs (word-boundary iteration with ASCII fast paths), sentence.rs (sentence-boundary iteration), and the large generated tables.rs (3,341 lines of Unicode property lookup tables), all tied together by a single UnicodeSegmentation trait defined in lib.rs that is implemented for &str. Boundary algorithms walk the string using state machines driven by property-class lookups rather than per-language rule sets, matching the UAX #29 specification directly. Tech Stack: Pure Rust, 2018 edition, MSRV 1.85.0, with zero runtime dependencies — dev-dependencies are limited to quickcheck, criterion, and proptest for testing/benchmarking only, and the crate has no_std support (no libstd requirement) via conditional compilation. Four Criterion benchmark suites (chars, words, word_bounds, unicode_word_indices) track performance regressions across releases. Code Quality: The tests/ directory includes a test.rs harness plus a testdata/ directory of official Unicode conformance test files, meaning correctness is validated against the same reference data Unicode publishes for UAX #29 compliance, not just hand-written unit tests; property-based tests via quickcheck/proptest add fuzzing-style coverage for edge cases like malformed or boundary-adjacent code points. API Design: The trait-based design lets consumers call .graphemes(true), .unicode_words(), etc. directly on any &str with a single use unicode_segmentation::UnicodeSegmentation; import, requiring no setup or configuration — an ergonomic, idiomatic-Rust extension-trait pattern that keeps the public surface minimal (four iterator methods plus the GraphemeCursor type) while doing this correctly is normally quite involved.