Chonkie

Chonkie is a lightweight Python library for splitting text into chunks for retrieval-augmented-generation (RAG) pipelines. It ships a wide range of chunking strategies — token, sentence, recursive, semantic, code-aware, table-aware, and a neural/“slumber” chunker — behind a consistent, minimal-install API, so teams can pick the chunking approach that fits their content without pulling in a heavyweight NLP stack.

Beyond chunking itself, Chonkie provides an end-to-end ingestion pipeline: fetchers to pull in source documents, a “chef” module to pre-process them, refinery steps to clean and merge chunks, embeddings and “handshakes” to hand chunks off to 32+ vector database and embedding integrations, and a cloud option (Chonkie Cloud) for teams that want managed chunking. It’s built for multilingual (56-language) content and designed to be installed piecemeal — only the chunkers and integrations you actually use.

What You Get

• 10+ chunking strategies (token, sentence, recursive, semantic, code-aware, table-aware, late, neural/slumber) behind one consistent callable interface
• A Pipeline abstraction to chain fetching, chunking, refining, and embedding/shipping to a vector store in one declarative flow
• 32+ integrations with popular vector databases, embedding providers, and RAG tooling, installable piecemeal via extras
• Multilingual chunking support out of the box across 56 languages
• A CLI and an optional Chonkie Cloud service for managed/hosted chunking
• Genie and handshake modules for connecting chunked output to downstream LLM and storage systems

Common Use Cases

• Preprocessing documents into semantically coherent chunks before embedding for a RAG retrieval pipeline
• Chunking source code by function/class boundaries for code-search or code-assistant RAG systems
• Building an end-to-end ingestion pipeline (fetch → chunk → refine → embed → store) for a document Q&A product
• Chunking multilingual content consistently across a corpus spanning many languages
• Swapping chunking strategies (token vs. semantic vs. recursive) to A/B test retrieval quality without changing pipeline code

Under The Hood

Architecture: Chunkers live under src/chonkie/chunker/, each subclassing a common base.py interface so RecursiveChunker, SemanticChunker, CodeChunker, TableChunker, and others are interchangeable at the call site. src/chonkie/pipeline/ composes these with src/chonkie/fetcher/ (source ingestion), src/chonkie/chef/ (pre-processing), src/chonkie/refinery/ (chunk post-processing/merging), src/chonkie/embeddings/ and src/chonkie/handshakes/ (vector-store integrations) into a single declarative flow, with src/chonkie/cloud/ and src/chonkie/api/ providing an optional hosted-service path and src/chonkie/cli/ a command-line entry point.

Tech Stack: Pure Python (requires-python >= 3.10), packaged with setuptools/wheel, dependency-locked with uv.lock, and explicitly designed around a “minimum installs” philosophy — the base package is lightweight and each chunker/integration is an opt-in extra rather than a bundled dependency, keeping the reported package size to ~505KB.

Code Quality: 93 test files, Codecov-tracked coverage, and a BENCHMARKS.md documenting chunking-speed comparisons against other libraries, indicating a maintainer focus on both correctness and the project’s core performance claim. py.typed marks the package for typed consumption, and a Dockerfile/docker-compose.yml support containerized dev/test workflows.

API Design: The core usage pattern — from chonkie import RecursiveChunker; chunker = RecursiveChunker(); chunks = chunker(text) — is deliberately minimal, and every chunker shares this same call shape, so switching strategies is a one-line change. The optional Pipeline API adds composability for teams that need a full ingestion flow without forcing that complexity on users who just want to chunk text. Extensive docs at docs.chonkie.ai and an llms.txt file for LLM-assisted onboarding round out the developer experience.