spacy-pkuseg

spacy-pkuseg is Explosion’s fork of the academic pkuseg-python toolkit for Chinese word segmentation, repackaged to install cleanly as a pip dependency and serialize well for use inside spaCy pipelines. The underlying segmentation algorithm and pretrained models are unmodified from the original research project; this fork focuses purely on packaging, wheel distribution, and compatibility fixes.

Unlike general-purpose Chinese tokenizers, pkuseg ships multiple domain-specific pretrained models (news, web/social media, medicine, tourism, and a mixed-domain default), letting callers pick a model tuned to their text’s domain for higher segmentation accuracy, plus support for training custom models on labeled data and part-of-speech tagging.

What You Get

• Domain-specific pretrained segmentation models (news, web, medicine, tourism, mixed/default)
• Cython-accelerated inference and feature extraction for fast segmentation on large corpora
• Support for training custom segmentation models on your own labeled data
• Optional part-of-speech tagging alongside word segmentation
• A packaging fork optimized for pip installation and spaCy pipeline integration, distributed as prebuilt wheels
• A CLI/Python API usable independently of spaCy for standalone segmentation tasks

Common Use Cases

• Tokenizing Chinese text as a preprocessing step in a spaCy NLP pipeline
• Segmenting domain-specific text (news articles, medical records, travel reviews) with a matching pretrained model
• Building custom Chinese segmentation models for a specialized vocabulary or domain
• Adding part-of-speech tagging to a Chinese text-processing pipeline

Under The Hood

Architecture — The package’s Python layer (spacy_pkuseg/) dispatches segmentation calls into Cython extensions compiled from inference.pyx and feature_extractor.pyx (plus a postag/feature_extractor.pyx for tagging), which implement the actual sequence-labeling inference used by pkuseg’s segmentation models; pretrained model weights and dictionaries ship under spacy_pkuseg/models and spacy_pkuseg/dicts. Tech Stack — Python plus Cython/C++ extension modules built via setuptools/cibuildwheel, depending on NumPy for numerical operations; prebuilt wheels are published per-platform so most users never compile from source. Code Quality — Tests live under spacy_pkuseg/tests; as a packaging fork of an unmaintained upstream research project, the code itself is largely frozen (health signals show low recent commit activity), so quality here mostly reflects the stability of the original pkuseg algorithm rather than active engineering churn. API Design — The API is minimal by design — instantiate a pkuseg object with a model name and call .cut(text) — which keeps the learning curve very low, though documentation is primarily in Chinese with a separate English README, and the library-vs-fork relationship (spacy-pkuseg vs. upstream pkuseg-python) can be a source of confusion for new users.