bstr

bstr is a widely-used Rust crate that provides extension traits for &[u8] and Vec<u8>, enabling their use as byte strings. Unlike the standard library’s str and String, byte strings are only conventionally UTF-8 — they may be fully valid, partially valid, or contain arbitrary bytes — which makes bstr ideal for text processing that must tolerate invalid encodings.

On top of raw bytes, bstr layers a rich, Unicode-aware string API: substring search, splitting, line and grapheme iteration, case conversion, and lossy UTF-8 decoding. Downloaded hundreds of millions of times, it underpins tooling like ripgrep where handling real-world, possibly-malformed text is essential.

What You Get

• Extension traits (ByteSlice, ByteVec) adding string methods to &[u8] and Vec<u8>.
• BStr and BString types for byte strings that display and debug like text.
• Unicode-aware iteration over characters, grapheme clusters, words, sentences, and lines.
• Fast substring search plus splitting, trimming, replacing, and case conversion.
• Lossy UTF-8 decoding utilities and optional serde and std integration via feature flags.

Common Use Cases

• Processing file or network data that is conventionally text but may contain invalid UTF-8.
• Building search and text tools (like ripgrep) that must not choke on malformed encodings.
• Iterating over graphemes, words, or lines with correct Unicode segmentation.
• Performing high-performance substring search and splitting over raw bytes.

Under The Hood

Architecture — bstr is organized around extension traits in ext_slice.rs and ext_vec.rs that attach string methods to [u8] and Vec<u8>, plus the BStr/BString wrapper types in impls.rs. Unicode segmentation lives under src/unicode/ (grapheme, word, sentence breaking) backed by generated DFA tables, while utf8.rs handles lossy decoding, ascii.rs provides fast ASCII paths, escape_bytes.rs handles debug rendering, and io.rs adds byte-string I/O helpers. Substring search delegates to the memchr crate for SIMD-accelerated scanning.

Tech Stack — Rust with optional std, alloc, unicode, and serde feature flags for no_std and integration scenarios. Core dependencies include memchr for fast searching and serde (optional). Dual-licensed MIT OR Apache-2.0. A small amount of shell tooling regenerates Unicode tables.

Code Quality — Testing is extensive and mature: dozens of #[test] blocks span utf8.rs, impls.rs, ascii.rs, ext_vec.rs, and the unicode modules, complemented by property-style and table-driven tests. The crate is authored and maintained by Andrew Gallant (BurntSushi) with 45 contributors and a long, active history, and it is battle-tested as a dependency of ripgrep and hundreds of millions of downstream builds.

API Design — The API mirrors the standard str/String surface closely, so existing Rust knowledge transfers directly — you reach for familiar methods like split, contains, find, to_lowercase, and lines, but on bytes. The extension-trait approach means you opt in by importing a trait; documentation is thorough and example-rich, keeping the learning curve low despite the depth of Unicode functionality underneath.