lindera-core

lindera-core is the foundational crate of the Lindera morphological analysis project, providing bincode/serde-serializable dictionary structures (PrefixDict, ConnectionCostMatrix, CharacterDefinitions, UnknownDictionary, WordEntry) and the Viterbi-lattice tokenization algorithm used to segment Japanese, Chinese, and Korean text into words. It underlies Lindera’s IPADIC-, UniDic-, ko-dic-, and CC-CEDICT-based tokenizers, exposing byte-level prefix-dictionary lookups and a lattice builder that resolves the lowest-cost path through candidate word boundaries via connection-cost tables compiled from dictionary source data.

As of Lindera 1.0 (mid-2025) the crate’s responsibilities were folded into lindera-dictionary and the umbrella lindera crate, and lindera-core is no longer part of the active workspace — its last published release was 0.32.3 in March 2025. It remains installable from crates.io for projects still on Lindera’s 0.3x line, and reading it is a direct way to understand the lattice/dictionary core that the current lindera and lindera-dictionary crates grew out of.

What You Get

• PrefixDict lookups - a double-array-backed prefix dictionary (via the yada crate) for fast prefix() lookups against both system and user dictionaries.
• Viterbi lattice tokenizer - builds a lattice of candidate word edges over input text and finds the minimum-cost segmentation path using a connection-cost matrix.
• Unknown-word handling - CharacterDefinitions category lookups plus an UnknownDictionary that synthesize candidate edges for text not found in any loaded dictionary.
• Bincode-serializable dictionary types - Dictionary and UserDictionary structs that (de)serialize compiled dictionary binaries via serde + bincode.
• Typed error handling - a LinderaError/LinderaErrorKind pair (via thiserror) that tags failures by cause (Io, Decode, DictionaryLoadError, etc.) instead of surfacing raw errors.

Common Use Cases

• Building a custom dictionary-backed tokenizer - a team assembling a new CJK dictionary crate implements the DictionaryBuilder trait against lindera-core’s types to plug into the existing lattice/tokenization pipeline.
• Pinning to Lindera’s 0.3x line - a project already depending on Lindera 0.3x’s public API keeps using lindera-core directly for dictionary/lattice types rather than migrating to the merged lindera-dictionary crate.
• Understanding Lindera’s segmentation internals - a contributor or researcher reads lindera-core’s Viterbi lattice and connection-cost-matrix code to understand how the higher-level lindera crate actually segments text.
• Embedding pre-compiled dictionary binaries - an application uses ConnectionCostMatrix::load_static and Dictionary’s Cow<‘static, [u8]> fields to ship a dictionary compiled directly into the binary via include_bytes! rather than loaded from disk at runtime.

Under The Hood

Architecture The crate centers on a byte-offset-indexed Lattice (viterbi.rs): Lattice::set_text scans each byte offset of the input, queries a PrefixDict (and an optional user PrefixDict) for KNOWN word edges, and where no dictionary word starts falls back to CharacterDefinitions category lookups plus UnknownDictionary::lookup_word_ids to synthesize UNKNOWN edges; calculate_path_costs then runs a single forward dynamic-programming pass, combining each edge’s word_cost with ConnectionCostMatrix.cost(left.right_id, right.left_id) and a Mode::penalty_cost to pick the minimum-cost predecessor per edge, and tokens_offset backtracks from the terminal EOS_NODE via each edge’s left_edge pointer to recover the final segmentation. Module boundaries are clean: character_definition.rs, connection.rs, dictionary.rs, prefix_dict.rs, unknown_dictionary.rs and word_entry.rs each own one data structure, while dictionary_builder.rs defines the DictionaryBuilder trait that concrete dictionary crates (IPADIC, UniDic, ko-dic, CC-CEDICT) implement to compile their own source data into this crate’s binary formats.

Tech Stack Pure Rust (edition 2021) with no async runtime and no I/O framework — this is a leaf computational library meant to be embedded by higher-level dictionary and tokenizer crates rather than run standalone. Dependencies are narrowly scoped to the job: anyhow and thiserror for error typing, bincode and serde (derive) for dictionary (de)serialization, byteorder for little-endian binary decoding, encoding_rs for character-set handling, once_cell for lazy statics, and yada for the double-array trie backing PrefixDict.

Code Quality Across roughly a dozen source files and just over a thousand lines, only two files (word_entry.rs and character_definition.rs) carry #[test] blocks, and the core lattice, dictionary, and connection-matrix logic has no direct unit coverage of its own — correctness leans on integration tests in the sibling lindera crate that consumes it. Error handling is centralized through LinderaError/LinderaErrorKind with a with_error/add_context builder pattern rather than ad hoc panics, though the lattice code itself still relies on array-index access and unwrap() on malformed offsets rather than propagating a Result. Naming is consistent snake_case throughout with light doc-comments, but no CI configuration ships inside the standalone crate and there are no crate-level usage examples.

API Design The public surface is deliberately low-level and mechanical: a caller constructs a Lattice, calls set_text with borrowed dictionary, user-dictionary, character-definition, and unknown-dictionary references plus a search Mode, then calls calculate_path_costs and tokens_offset to get back a Vec<(usize, WordId)>. There is no builder/config struct, set_text and calculate_path_costs return (), and there are no doctests or example files in the crate — using it correctly requires already understanding Lindera’s lattice model, which is exactly why the project later wrapped this into the more ergonomic Tokenizer API in lindera-dictionary and lindera once the workspace matured.