jieba-rs

jieba-rs is a Rust port of the widely used Jieba Chinese text segmentation library, providing word splitting, part-of-speech tagging, and keyword extraction for CJK text. It ships an embedded dictionary by default so segmentation works out of the box with no external data files, while optional Cargo features add TF-IDF and TextRank keyword extraction for downstream NLP or search workflows.

Built as a Cargo workspace, the core jieba-rs crate exposes cut, cut_all, cut_for_search, tokenize, and tag APIs mirroring the original Python Jieba’s interface, plus a C-compatible API crate (capi) that lets non-Rust projects link against the same segmentation engine. The project has spawned bindings for Node.js, PHP, Python, R, WebAssembly, and Emacs, plus integrations with the Tantivy search engine, making it a common building block anywhere Chinese text needs to be tokenized.

What You Get

• Dictionary-based segmentation via a prefix-tree DAG (cut, cut_all, cut_for_search) with HMM-based recognition for out-of-vocabulary words
• Part-of-speech tagging (tag) using the ported Jieba POS tag set
• Optional TF-IDF and TextRank keyword extractors gated behind Cargo features
• Custom dictionary loading and runtime word/frequency overrides via load_dict/add_word
• A C-compatible capi crate for linking the segmenter from non-Rust languages

Common Use Cases

• Tokenizing Chinese text before indexing it in a search engine (used by the tantivy-jieba and cang-jie integrations)
• Chinese NLP preprocessing pipelines that need word/POS boundaries before downstream models
• Extracting representative keywords from Chinese documents via TF-IDF or TextRank
• Powering Chinese segmentation in non-Rust apps through the capi C bindings or the WASM build

Under The Hood

Architecture The project is a four-member Cargo workspace: the core jieba crate holds the segmentation engine, jieba-macros is a proc-macro crate used at build time, capi wraps the core crate in a C ABI for FFI consumers, and examples/weicheng is a runnable example binary. Inside the core crate, lib.rs orchestrates a double-array trie (cedarwood) built into a directed acyclic graph over the embedded dictionary for known-word segmentation, hmm.rs implements a Hidden Markov Model with Viterbi decoding (via thread-local HmmContext, avoiding global mutable state) for unknown-word recognition, posseg.rs runs a separate HMM for part-of-speech tagging, and sparse_dag.rs holds the DAG data structure. Keyword extraction lives in its own keywords module with tfidf.rs and textrank.rs as pluggable, feature-gated implementations of a shared KeywordExtract trait. The separation between dictionary/DAG construction, HMM inference, and keyword extraction is clean, with each concern in its own module.

Tech Stack Rust edition 2024 across the workspace. Core dependencies are cedarwood (double-array trie for the dictionary), rustc-hash (FxHashMap for fast hashing), regex, phf (perfect hash for static lookups), include-flate (compresses and embeds the ~5MB dictionary/IDF/POS data files directly into the binary), and bytecount (SIMD-accelerated character counting). Dev dependencies include codspeed-criterion-compat for benchmarking, expect-test for snapshot-style assertions, wasm-bindgen-test for WASM test coverage, rayon for parallel benchmarks, and tikv-jemallocator on Unix for benchmark allocator tuning.

Code Quality 33 unit tests live directly under jieba/src, alongside multiple runnable doctests embedded in lib.rs’s module documentation. CI (GitHub Actions) checks the crate under every feature combination (no-default-features, default, tfidf, textrank) across Ubuntu, macOS, and Windows, runs the full test suite with cargo test --workspace --all-features --all --benches, tracks coverage via cargo-llvm-cov uploaded to Codecov, and enforces formatting with cargo fmt --check. Errors are modeled explicitly through an Error enum (Io, InvalidDictEntry, InvalidHmmModel) implementing std::error::Error rather than panicking or swallowing failures, and fallible constructors return Result. Naming is consistent, idiomatic Rust throughout.

API Design The public API is intentionally small: Jieba::new() gives a zero-config instance backed by the embedded dictionary, and cut/cut_all/cut_for_search/tokenize/tag mirror the original Python Jieba’s method names, so developers already familiar with Jieba transfer knowledge directly. Keyword extraction and the embedded dictionary are opt-in/opt-out via additive Cargo features (tfidf, textrank, default-dict), keeping the default build lean. The module-level rustdoc includes runnable examples for the default segmentation, TF-IDF, and TextRank paths, so cargo doc output is usable on its own without consulting the README.