esaxx-rs

esaxx-rs is a small Rust wrapper around the esaxx suffix-array C++ library from Google’s SentencePiece. It builds a suffix tree / suffix array over a string and lets you enumerate its distinct substrings together with their occurrence counts, a building block for text tokenization and substring frequency analysis.

Originally extracted from SentencePiece and used by Hugging Face’s tokenizers crate, it exposes a single ergonomic suffix() entry point that returns an iterator over substrings and their frequencies. It can build against the bundled C++ implementation (default) or a pure-Rust SAIS fallback.

What You Get

• A single suffix() function that builds a suffix array/tree over a string
• An iterator yielding distinct substrings paired with their occurrence counts
• Bindings to SentencePiece’s battle-tested esaxx C++ implementation
• An optional pure-Rust SAIS implementation as an alternative to the C++ backend
• A tiny, dependency-light API proven in production by Hugging Face tokenizers

Common Use Cases

• Training subword tokenizers that need frequent-substring statistics
• Finding and counting repeated substrings within large text corpora
• Building suffix-array-backed text indexing or analysis tooling in Rust

Under The Hood

Architecture - The public surface in src/lib.rs wraps a C++ core: src/esaxx.cpp (with headers esa.hxx/sais.hxx) implements the suffix-array construction, bridged to Rust via a cc-compiled build. Rust-side counterparts src/esa.rs and src/sais.rs provide a pure-Rust SAIS path, while src/types.rs defines the shared suffix-iterator types.

Tech Stack - Rust (edition 2018) with essentially no runtime dependencies; the default cpp feature uses the cc build dependency to compile the bundled C++ sources. Criterion is used only for benchmarks. The default = ["cpp"] feature can be disabled to fall back to the Rust SAIS implementation.

Code Quality - Tests are colocated in src/lib.rs and src/sais.rs, and a Criterion benchmark (bench_suffix) plus an example binary exercise the API. Because the core algorithm is inherited from SentencePiece, it is widely validated in real-world tokenizer pipelines.

API Design - The API is intentionally minimal, one suffix() call returning an iterator, matching the documented README example almost exactly. This keeps the learning curve low despite the underlying complexity of suffix-array construction, and docs.rs documentation covers the iterator semantics.