text-splitter

text-splitter breaks long documents into smaller chunks that respect semantic boundaries — grapheme clusters, words, sentences, and newlines — while staying as close as possible to a target chunk size measured in characters or tokens. It is built for the common LLM workflow of fitting arbitrarily long documents into a model’s limited context window without cutting sentences or paragraphs mid-thought.

Beyond plain text, the crate ships a MarkdownSplitter that understands CommonMark structure (headings, code blocks, lists) and a CodeSplitter that uses tree-sitter grammars to split source code along syntax-tree boundaries. Chunk size can be measured by raw character count or by plugging in a Hugging Face tokenizers or tiktoken-rs tokenizer, so chunk boundaries can be made to match a specific model’s actual token limits.

What You Get

• TextSplitter for plain text, chunking along Unicode grapheme/word/sentence/newline boundaries
• MarkdownSplitter (via the markdown feature) that respects CommonMark structure such as headings and code blocks
• CodeSplitter (via the code feature) that splits source files along tree-sitter syntax-tree boundaries
• Pluggable chunk-size measurement: raw character count, a range of sizes, a Hugging Face tokenizers::Tokenizer, or a tiktoken-rs tokenizer
• Python bindings (semantic-text-splitter on PyPI) sharing the same Rust core

Common Use Cases

• Chunking documents before embedding them for a RAG (retrieval-augmented generation) pipeline
• Splitting long source text to fit within an LLM’s token-limited context window without breaking mid-sentence
• Preprocessing Markdown documentation into semantically coherent chunks for search indexing
• Splitting source code files along syntax-aware boundaries for code-search or code-LLM tooling

Under The Hood

Architecture: The crate is organized around a shared ChunkConfig/ChunkSizer abstraction in src/chunk_size.rs (915 lines) that decouples “how big is this chunk” from “where can I split,” letting TextSplitter, MarkdownSplitter, and CodeSplitter (in src/splitter.rs, 791 lines, plus the src/splitter/ submodule) reuse the same greedy merge algorithm: split at increasing semantic levels, then greedily merge neighboring sections up to the configured size without crossing a higher-level semantic boundary. src/trim.rs handles whitespace trimming behavior at chunk edges, and bindings/ hosts the separate Python binding crate as a workspace member.

Tech Stack: Rust 2021 edition (MSRV 1.86), leaning on Unicode’s icu_segmenter/icu_provider crates for grapheme/word/sentence boundary detection (replacing hand-rolled sentence-splitting heuristics), pulldown-cmark for CommonMark parsing (optional markdown feature), tree-sitter for code parsing (optional code feature), and optional tiktoken-rs/tokenizers integrations for token-aware sizing. ahash, itertools, either, and strum support internal data structures and enum dispatch.

Code Quality: The tests/ directory includes dedicated suites for text, Markdown, code, and tokenizer-based splitting (text_splitter.rs, markdown.rs, code.rs, tokenizers/), plus an insta-based snapshot-testing setup (tests/snapshots/) that pins exact chunking output against a large corpus of tests/inputs/ fixtures — a strong signal for a library whose correctness hinges on exact boundary behavior. A divan-based benchmark suite (benches/chunk_size) tracks performance, and the workspace enables clippy::pedantic and clippy::cargo lints project-wide.

API Design: All three splitter types (TextSplitter, MarkdownSplitter, CodeSplitter) share the same new(size) / .chunks(text) call pattern, so switching from plain-text to Markdown- or code-aware splitting is a type substitution rather than a new API to learn. Chunk sizing is similarly uniform — pass a plain integer, a range, or any type implementing ChunkSizer (including third-party tokenizers) — which keeps the common case (TextSplitter::new(1000)) a one-liner while still supporting token-exact sizing for production LLM pipelines.