unicode-general-category

unicode-general-category is a small Rust crate that exposes get_general_category, a constant-time lookup function returning the Unicode General Category property for any char. It targets no_std environments and embedded targets, using a two-level lookup table compiled at build time from Unicode Character Database (UCD) data covering Unicode 16.0.

The build script converts a flat table of category ranges into a block-based structure that deduplicates repeated runs, trading a small amount of table size (about 45KiB) for lookup speed reported to be 5-10x faster than the typical binary search approach used by similar crates. The generated table and enum are produced with the yeslogic-ucd-generate tool, keeping the crate’s data in sync with each new Unicode release.

What You Get

• get_general_category(char) -> GeneralCategory, an O(1) lookup function backed by a precompiled two-level table
• A GeneralCategory enum covering all 30 Unicode general category values, each with an abbreviation() method (e.g. Lu, Ll, Nd)
• no_std compatibility, verified in CI against the thumbv7em-none-eabihf embedded target
• Data generated from Unicode 16.0 via the yeslogic-ucd-generate tool, keeping category data current with the latest Unicode Character Database

Common Use Cases

• Text processing pipelines that need to classify characters (letters vs. punctuation vs. symbols) without pulling in a full Unicode library
• Font shaping and text layout engines that need per-codepoint category data at parse time
• Embedded or WASM projects that need Unicode classification but cannot depend on the standard library
• Parsers and lexers that branch on Unicode categories (e.g. treating all ‘Number’ categories the same way)

Under The Hood

Architecture The crate has a minimal, single-purpose architecture: src/lib.rs is the public entry point (no_std, re-exporting category::get_general_category and tables::GeneralCategory), src/category.rs implements the actual lookup logic by including a build-script-generated file (env!(“OUT_DIR”)/category.rs) that defines BLOCK_SIZE, CATEGORY_BLOCKS, CATEGORY_BLOCK_OFFSETS and LAST_CODEPOINT, and src/tables.rs holds the raw, unprocessed Unicode Character Database ranges (GENERAL_CATEGORY, a large const array of (start, end, category) tuples) consumed only by build.rs. Data flow is strictly one-directional and compile-time: build.rs re-parses src/tables.rs at build time (via a #[path] inclusion), compresses the flat range table into a two-level block-indexed table, and writes the generated Rust source into OUT_DIR, which category.rs then includes verbatim — the “real” implementation of get_general_category doesn’t exist in the crate’s own source tree at all, only its generator does. This build-script/codegen split keeps the runtime crate to a single array-indexing operation with no parsing or search, at the cost of tables.rs being large and effectively write-only, regenerated exclusively via the Makefile’s tables target and the external yeslogic-ucd-generate tool. There is no plugin surface; changing the core abstraction (the two-level block scheme in build.rs) would require re-deriving its shift/mask constants and regenerating every downstream table.

Tech Stack Pure, zero-dependency Rust (edition 2018) with an empty dependencies table in Cargo.toml. The only tooling is the build.rs build script (std-only) that runs at compile time to generate the lookup table from src/tables.rs, plus a Makefile target that regenerates src/tables.rs itself using the external, separately-maintained yeslogic-ucd-generate CLI against a pinned Unicode Character Database version, followed by cargo fmt. CI tests against Rust beta, stable, and a deliberately old MSRV pin on Ubuntu, runs format checks and cargo test, and additionally cross-compiles for the thumbv7em-none-eabihf embedded target to enforce no_std compatibility. This is a library crate published to crates.io with docs.rs integration — no web framework, database, or deployment target involved.

Code Quality Tests live inline under a cfg(test) module using Rust’s built-in test/assert_eq! macros — no external testing framework, consistent with the crate’s zero-dependency philosophy. Coverage is a handful of spot-checks against specific characters rather than exhaustive range verification, but CI runs a format check and both a standard test run and an embedded-target build on every push and pull request across multiple Rust channels, which offsets some of the confidence a fuller test suite would otherwise provide. Naming follows idiomatic Rust conventions, and the public API surface is small enough that type safety is straightforward — a single function taking a char and returning a plain enum, no Result/Option, no documented panics. No linting step is visible in CI beyond formatting.

What Makes It Unique Two-level, block-indexed lookup tables for Unicode property data are a known technique, not a new algorithm class, but the execution here is deliberate: the build script derives its shift amount from the block size directly and deduplicates identical blocks before appending new ones, and the README documents concrete tradeoffs (table size, measured speedup versus typical binary-search approaches) rather than asserting them without evidence. The real distinguishing choice is scope discipline — covering exactly one Unicode property (General_Category) rather than the full gamut handled by broader Unicode crates, with no_std and embedded-target support verified in CI, which larger Unicode-property crates often skip.