jieba

jieba (Chinese for “to stutter”) is a Python library for segmenting Chinese text into words — a prerequisite step for Chinese NLP since, unlike space-delimited languages, Chinese text has no explicit word boundaries. It builds a directed acyclic graph of all possible word combinations from a prefix dictionary, then uses dynamic programming to find the maximum-probability segmentation path based on word frequency, falling back to an HMM/Viterbi model for previously unseen words.

The library supports four cut modes — precise (best for text analysis), full (fastest, scans every possible word), search-engine (re-segments long words for better recall in search indexing), and an optional PaddlePaddle deep-learning mode — plus part-of-speech tagging, keyword extraction (TF-IDF and TextRank), custom user dictionaries, and traditional Chinese support. Despite low recent commit activity, it remains the most widely used Chinese segmentation library in the Python ecosystem, with ports and bindings in numerous other languages.

What You Get

• Four segmentation modes: precise, full, search-engine, and an optional PaddlePaddle deep-learning mode
• Part-of-speech tagging via the jieba.posseg module
• Keyword extraction using TF-IDF and TextRank algorithms via jieba.analyse
• Custom user dictionary support for domain-specific vocabulary and traditional Chinese text
• HMM/Viterbi-based new-word recognition for terms not present in the built-in dictionary

Common Use Cases

• Preprocessing Chinese text for search-engine indexing, where the search-mode cut improves recall on long words
• Tokenizing Chinese corpora before feeding them into downstream NLP models (topic modeling, classification, embeddings)
• Extracting keywords/tags from Chinese articles or documents via TF-IDF or TextRank
• Adding custom terminology (brand names, technical jargon) to segmentation via user-supplied dictionaries

Under The Hood

Architecture - The core jieba.cut() pipeline builds a prefix-tree-backed dictionary (dict.txt), constructs a directed acyclic graph of all valid word segmentations of the input using that dictionary, then runs dynamic programming to find the maximum-frequency-weighted path through the DAG; the finalseg submodule applies a pretrained HMM with Viterbi decoding to catch out-of-dictionary words, while posseg and analyse layer POS tagging and TF-IDF/TextRank keyword extraction on top of the same tokenizer. Tech Stack - Pure Python (2/3 compatible historically) with no required third-party runtime dependencies for core segmentation; the optional paddle mode adds a dependency on paddlepaddle-tiny for a bidirectional-GRU sequence-labeling model. Code Quality - 36 test scripts exercise segmentation modes, POS tagging, and keyword extraction against example corpora; the codebase predates modern type-hint conventions and has seen minimal structural change in recent years, reflecting its status as a stable, feature-complete utility rather than actively evolving software. API Design - The primary API (jieba.cut, jieba.lcut, jieba.cut_for_search) is a small set of generator/list-returning functions with intuitive boolean flags (cut_all, HMM, use_paddle), requiring no configuration to get first results, which has made it the default choice for Chinese tokenization in Python despite limited recent maintenance.