gse

gse is a Go implementation of jieba-style word segmentation, built to tokenize Chinese, Japanese, English, and other languages inside Go applications without shelling out to a Python NLP stack. It represents its dictionary as a double-array trie (via the sibling vcaesar/cedar package) and layers multiple cutting strategies on top: a shortest-path DAG segmenter for known words, an HMM Viterbi decoder for out-of-vocabulary sequences, and a search-engine mode that emits overlapping sub-tokens for indexing.

Beyond plain segmentation, gse ships part-of-speech tagging, stop-word trimming, TF-IDF lookups, user/embedded dictionary loading (including Go 1.16+ //go:embed support), and Traditional Chinese handling. It integrates with Elasticsearch and Bleve through companion packages, and includes an optional JSON-RPC server under tools/server for using the segmenter outside of a Go process. Experimental CRF and neural-network (gonn) subpackages exist alongside the stable dictionary/HMM core for teams that want to extend the statistical layer.

Because the whole thing is a plain importable Go module with a minimal dependency footprint, it drops into search indexing, log/text analytics, and content-processing pipelines that need fast multilingual tokenization without an external NLP service.

What You Get

• A Segmenter type with Cut, CutSearch, CutAll, and CutDAG methods covering accurate, search-engine, full, and HMM-assisted segmentation modes
• A double-array trie dictionary (via vcaesar/cedar) supporting embedded (//go:embed), file-based, and in-memory dictionary loading, plus custom user dictionaries
• HMM Viterbi-based cutting for out-of-vocabulary words, with part-of-speech tagging via the hmm/pos subpackage
• Stop-word trimming, TF-IDF term lookup, and Traditional Chinese support built into the core segmenter
• An optional JSON-RPC server (tools/server) for running gse as a standalone segmentation service outside a Go process

Common Use Cases

• Tokenizing Chinese, Japanese, or English text before indexing it into Elasticsearch or Bleve for full-text search
• Preprocessing multilingual log or document text for keyword extraction and TF-IDF scoring
• Adding Chinese/Japanese word segmentation to a Go-based content pipeline without calling out to a Python NLP service
• Building custom NLP tooling on top of POS tagging and stop-word filtering for downstream text analytics

Under The Hood

Architecture The core Segmenter type (segmenter.go, gse.go) wraps a Dictionary (dictionary.go) backed by a double-array trie from the sibling vcaesar/cedar package, with dictionary loading and Go 1.16+ embed support split into dict_util.go, dict_1.16.go, and dict_embed.go. Segmentation itself is layered: dag.go implements shortest-path DAG cutting over dictionary-known tokens, while the hmm package (hmm_seg.go, viterbi.go, prob_emit.go/prob_trans.go) provides an independent HMM Viterbi decoder for out-of-vocabulary sequences, with hmm/pos reusing the same Viterbi machinery for part-of-speech tagging. Statistical extensions - crf/ (CRF) and gonn/ (CNN/RNN) - are kept as loosely-coupled sibling packages rather than wired into the core Segmenter, so the stable dictionary/DAG/HMM path doesn’t depend on the more experimental statistical layers; an optional JSON-RPC server under tools/server exposes the same Segmenter over the network. Changing the underlying trie representation would ripple through dag.go, seg_utils.go, and dict_util.go, since all three operate directly on the cedar-backed byte trie.

Tech Stack gse is a Go 1.25 module with a genuinely small dependency footprint - only github.com/vcaesar/cedar (the double-array trie) and github.com/vcaesar/tt (a test-assertion helper) in go.mod, no database, web framework, or ORM involved since this is a pure algorithmic library. Default dictionary data ships under data/ and can be loaded from disk or embedded directly into the binary via Go’s //go:embed directive. CI (.github/workflows/go.yml) builds and tests across a Go 1.25.x/1.26.x x macOS/Windows/Linux matrix via GitHub Actions, with a CircleCI badge and Codecov integration also present in the README.

Code Quality Tests exist at every layer that matters for correctness: gse_test.go, segmenter_test.go, token_test.go, and dict_1.16_test.go cover the public segmentation API and dictionary loading, gse_bm_test.go adds throughput benchmarks, and hmm/hmm_seg_test.go plus hmm/pos/pos_seg_test.go cover the statistical layer independently. Tests use the standard testing package with the lightweight vcaesar/tt assertion helper rather than testify. Error handling is explicit - functions like Dictionary.AddToken and Segmenter.LoadDict return error values instead of panicking - and public API naming is consistent (the Cut* family all returns []string). There’s no dedicated lint config or lint step in CI beyond go build/go test, which is a gap relative to a fully typed-and-linted setup.

API Design The public surface is a single exported Segmenter struct configured through public fields (AlphaNum, SkipPos, NotStop, MinTokenFreq, etc.) rather than a functional-options constructor, so getting started is a one-liner (gse.New(dictPath) or a zero-value Segmenter{} plus LoadDict()), at the cost of configuration happening via mutable struct state instead of an immutable builder. The Cut/CutSearch/CutAll/CutDAG method family is named consistently and each returns []string, though toggling HMM behavior through a hmm ...bool variadic argument is a more implicit pattern than a named boolean parameter would be. The README documents several concrete usage patterns - basic cutting, embedded dictionaries, custom dictionaries, and POS tagging - each backed by a runnable example under examples/, which lowers the barrier for the common cases even though some of the tuning flags are only discoverable by reading source.