lindera

Lindera is a multilingual morphological analysis library written in Rust, forked from kuromoji-rs, aimed at giving Rust applications an easy-to-install, dependency-free tokenizer for CJK languages. It exposes a Segmenter that loads a dictionary (embedded at compile time or read from disk, optionally memory-mapped), builds a Viterbi lattice per sentence, and returns tokens with surface text, byte offsets, and part-of-speech/reading details — the same output shape MeCab-derived tools produce, without requiring MeCab or any C dictionary library to build.

Beyond the core Rust crate, the project maintains a workspace of dictionary-specific crates (IPADIC, IPADIC NEologd, UniDic, ko-dic, CC-CEDICT, Jieba, SudachiDict) selected via Cargo features, a lindera-cli binary for downloading dictionaries and tokenizing text from the command line, and official bindings for Python, Node.js, Ruby, PHP, and WebAssembly built on the same segmentation core. It’s most commonly used as the tokenizer behind CJK-aware full-text search (e.g. as a custom analyzer for Tantivy-based search engines) and as a general-purpose text-segmentation step in NLP pipelines.

What You Get

• A pure-Rust Segmenter API for tokenizing Japanese, Chinese, and Korean text via Viterbi-lattice morphological analysis, with no C/C++ dictionary dependency to build.
• Six pluggable dictionary backends (IPADIC, IPADIC NEologd, UniDic, ko-dic, CC-CEDICT, Jieba) selectable via Cargo features, embeddable directly in the binary or loaded from disk with optional mmap.
• N-best segmentation (segment_nbest) returning ranked alternative tokenizations with costs, plus a SegmentWorker wrapper for reusing lattice/backtrace buffers across repeated calls in hot paths.
• User dictionary support (CSV or precompiled binary) for adding custom vocabulary on top of any system dictionary.
• First-class bindings for Python, Node.js, WebAssembly, Ruby, and PHP built from the same core, plus a lindera-cli binary for ad hoc tokenization and dictionary management.

Common Use Cases

• Tokenizing Japanese/Chinese/Korean text for search-engine indexing pipelines, such as a custom analyzer/tokenizer feeding a Tantivy-based full-text search index.
• Building NLP preprocessing steps (word segmentation, POS tagging) for downstream text classification, keyword extraction, or morphological normalization in Rust services.
• Embedding CJK tokenization directly into a WebAssembly bundle for in-browser search or text analysis without a server round-trip.
• Adding custom vocabulary (product names, jargon, proper nouns) to a base dictionary via user dictionaries for domain-specific segmentation accuracy.

Under The Hood

Architecture The core lindera crate re-exports its foundational types (Dictionary, Metadata, UserDictionary, Lattice, WordId, DictionaryBuilder, Schema) directly from the lower-level lindera-dictionary crate and layers a Segmenter on top that owns a Mode, a Dictionary, and an optional UserDictionary. Segmentation flows through sentence splitting (find_sentence_end, which scans for real delimiters \n, \t, 。, 、 and falls back to a documented 32KiB forced cut to bound lattice size, per issue #871), Viterbi lattice construction per sentence via lattice.set_text/set_text_nbest, backtrace into token offsets, and Token construction carrying byte offsets, position, and dictionary references. A SegmentWorker wraps a Segmenter to reuse the lattice and backtrace buffer across repeated calls for throughput-sensitive callers. Dictionary selection is driven by DictionaryKind/DictionaryScheme enums gated behind Cargo features, so consumers opt into exactly the language dictionaries and binary size they need, or load external dictionary files (optionally memory-mapped) at runtime.

Tech Stack The crate targets the Rust 2024 edition (rust-version 1.88) as part of a ~19-crate Cargo workspace. Core dependencies are anyhow for error handling, serde/serde_json for configuration and schema data, strum/strum_macros for enumerable dictionary kinds, percent-encoding and url for dictionary URI parsing, and log for tracing. Language-specific dictionary crates (lindera-ipadic, lindera-unidic, lindera-ko-dic, lindera-cc-cedict, lindera-jieba, lindera-sudachidict) live as separate workspace members pulled in as optional, feature-gated dependencies. The lower-level lindera-dictionary crate depends on crawdad (pinned to an exact version because the runtime walks its serialized byte layout directly) and daachorse for double-array trie structures. Dev dependencies include criterion for benchmarking and insta for snapshot testing. The workspace also builds bindings for Python, Node.js, Ruby, PHP, and WebAssembly, plus a standalone lindera-cli crate.

Code Quality The segmenter module carries extensive unit tests directly alongside the implementation: sentence-boundary edge cases, config parsing, full tokenization runs against the embedded IPADIC dictionary with exact token and part-of-speech assertions, user-dictionary CSV and precompiled-binary variants, and should_panic tests for malformed input. CI workflows run cargo fmt --all -- --check, per-crate test matrices across multiple toolchains and platforms (including Windows), plus separate periodic and regression suites. Comments throughout cite specific GitHub issue numbers when documenting non-obvious behavior (sentence-length capping, mmap defaults, whitespace-filter performance, unknown-word ladder semantics), indicating deliberate, tracked engineering rather than ad hoc fixes.

API Design The public surface is compact: Segmenter::new/from_config plus chainable builder methods (keep_whitespace, max_grouping_len, unknown_word_ladder) for progressive configuration, and segment/segment_nbest with _with_lattice/_with_buffers variants for callers who want to reuse allocations across repeated calls. Configuration is a plain serde_json::Value (SegmenterConfig), so callers can build it from a JSON literal, a file, or programmatically without a bespoke config type. Type aliases re-exported from lindera-dictionary keep one canonical vocabulary across the crate instead of duplicating types, and the DictionaryKind/DictionaryScheme enums give a self-describing way to enumerate supported dictionaries per compiled feature set. Compared to older MeCab-derived bindings, Lindera trades a C dependency for a pure-Rust implementation with embeddable or memory-mapped dictionaries, and documents a migration guide alongside design notes explaining why sentence length is capped.