Open Source Packages
Libraries, frameworks, tools, and SDKs used by the apps in this directory — scored the same way as the apps themselves.
All Packages
Category
Sort:Used by Most Apps
Tokenizers
Fast, production-grade tokenizer implementations for NLP and LLMs, written in Rust
micromark
The smallest 100% CommonMark-compliant markdown parser, with GFM and MDX support
tiktoken-rs
Fast Rust library for tokenizing text and counting tokens for OpenAI GPT models with tiktoken BPE.
lindera
A pure-Rust morphological analysis library for tokenizing Japanese, Chinese, and Korean text.
text-splitter
Split text into semantic chunks up to a desired size, by characters or tokens
charabia
Meilisearch's multilingual Rust tokenizer for segmenting and normalizing text
lindera-core
Core dictionary structures and Viterbi lattice algorithm powering Lindera's Japanese, Chinese, and Korean morphological analysis.
vaporetto
A fast, lightweight Rust tokenizer for Japanese text using pointwise prediction, with no_std support for embedded and WebAssembly targets.
html5gum
A WHATWG-spec-compliant HTML5 tokenizer for Rust with pluggable emitters, letting you trade convenience for near-zero-allocation performance.
lexer
A flex-inspired, regex-based JavaScript lexer with start conditions and longest-match scanning for building custom tokenizers.
