vaporetto

Vaporetto is a Rust crate for Japanese tokenization built around pointwise boundary prediction rather than sequence labeling, letting it score each character boundary independently for speed. It ships as a no_std-compatible core (alloc only) so it can run on embedded devices and in WebAssembly, alongside optional tag prediction for part-of-speech and pronunciation output.

The crate loads pre-trained models — either ones distributed by the Vaporetto project or converted from KyTea — and exposes a Predictor API that developers call directly from Rust code. Companion crates in the same workspace add composable pre/post-processing filters (vaporetto_rules) and a Tantivy tokenizer integration (vaporetto_tantivy), and benchmarks cited in the README put it roughly 8.7x faster than KyTea, the tokenizer it was designed to succeed.

What You Get

• A Predictor API for tokenizing UTF-8 Japanese text into space-separated tokens, plus optional part-of-speech and pronunciation tag prediction via the tag-prediction feature.
• Pre-trained models distributed separately (daac-tools/vaporetto-models) and a converter for existing KyTea models via the kytea feature and convert_kytea_model binary.
• A no_std/alloc-only build mode for embedded targets, demonstrated in the examples/embedded_device sample, plus a WebAssembly demo.
• Model manipulation tools to dump a model’s scoring dictionary to CSV, hand-edit boundary weights, and reload the patched model — useful for fixing specific mis-segmentations without retraining.
• Optional training support (train feature, backed by liblinear) for building custom models from fully or partially annotated corpora.

Common Use Cases

• Japanese NLP preprocessing pipelines - developers tokenizing Japanese text before feeding it into search indexing, text classification, or downstream NLP models.
• Full-text search tokenization - teams using vaporetto_tantivy to plug Japanese-aware tokenization directly into a Tantivy search index.
• Embedded and resource-constrained deployments - engineers running tokenization on microcontrollers or in WASM where a no_std, small-dependency tokenizer is required.
• Migrating off KyTea - projects with existing KyTea models converting them to Vaporetto’s format for a reported 8.7x speedup without retraining.

Under The Hood

Architecture The crate separates concerns into a Predictor (predictor.rs) that combines a CharScorer (char_scorer.rs) and TypeScorer (type_scorer.rs) — each scoring character boundaries from a different feature signal — over a Sentence (sentence.rs), the crate’s central data structure tracking raw text, character boundaries, and optional tags. Models are deserialized through model.rs/dict_model.rs/ngram_model.rs (and, behind the kytea feature, kytea_model.rs for importing legacy KyTea models), while trainer.rs and tag_trainer.rs (behind the train feature) wrap liblinear to fit new models from annotated corpora. The workspace keeps this core crate separate from vaporetto_rules (composable sentence/string filters) and vaporetto_tantivy (a Tantivy Tokenizer adapter), and from the CLI binaries (predict, train, evaluate, manipulate_model, convert_kytea_model) that consume the library — a change to Sentence’s boundary representation is the one abstraction that would ripple through nearly every module.

Tech Stack Rust (edition 2021, MSRV 1.88) with a deliberately small, mostly no_std-friendly dependency set: bincode 2.0 for model (de)serialization, daachorse 1.0 (a double-array Aho-Corasick automaton from the same authors) for dictionary pattern matching, and hashbrown 0.15 as the HashMap implementation; liblinear is pulled in only behind the optional train feature. Feature flags (std, alloc, cache-type-score, fix-weight-length, tag-prediction, charwise-pma, kytea, train, portable-simd) gate functionality so embedded and WASM builds can drop std entirely. The workspace ships pre-trained-model downloads, a docs.rs-hosted API reference, and a browser WASM demo alongside the crate itself.

Code Quality The crate carries 77 in-source unit tests concentrated in its scoring and sentence-handling modules, uses a typed VaporettoError enum rather than panicking on malformed input, and enforces #![deny(missing_docs)] so every public item must be documented — visible in the extensive rustdoc coverage across sentence.rs and predictor.rs. CI (.github/workflows/rust.yml) runs cargo check, cargo fmt —check, and cargo clippy with nursery lints across MSRV, stable, and nightly toolchains on every push, pull request, and a daily schedule, which is a stronger bar than most crates its size hold themselves to.

API Design The public surface is small and direct: construct a Predictor from a loaded Model, call predict on a mutable Sentence, then read tokens or (optionally) fill_tags for part-of-speech/pronunciation output — the doc-tested example in lib.rs is only a handful of lines end to end. The main friction is that models aren’t bundled with the crate; getting started requires a separate model download or a KyTea conversion step before the API becomes usable, and advanced workflows (dictionary patching, custom training) require reading CLI-oriented documentation rather than crate-level docs.