unicode-xid

unicode-xid is a small, focused Rust library that determines whether a char is a valid identifier for a parser or lexer according to the rules in Unicode Standard Annex #31 (UAX #31). It exposes the XID_Start and XID_Continue character properties, which define how programming-language identifiers may begin and continue in a Unicode-aware way.

Maintained by the unicode-rs project, the crate is a foundational dependency across the Rust ecosystem, used by tools like proc-macro2 and syn to validate identifier characters. It supports no_std builds so it can be used in embedded or constrained environments, drawing on generated Unicode property tables rather than any runtime data.

What You Get

• The UnicodeXID trait with is_xid_start and is_xid_continue methods on char
• Correct XID_Start and XID_Continue classification per Unicode Standard Annex #31
• Generated Unicode property tables covering the full character range
• no_std support for use in embedded or constrained environments
• A tiny, dependency-free API used widely across the Rust ecosystem

Common Use Cases

• Tokenizing identifiers in a parser or lexer for a programming language or DSL
• Validating that user-supplied names are legal Unicode identifiers
• Powering procedural-macro and syntax-tree crates that construct identifiers

Under The Hood

Architecture - The crate is deliberately minimal: lib.rs defines the UnicodeXID trait implemented for char with is_xid_start and is_xid_continue, both of which perform binary searches over the code-point range tables in tables.rs. Those tables are machine-generated from the Unicode Character Database by a Python script checked into the repository, so the classification stays in sync with the Unicode version it targets.

Tech Stack - Pure Rust with zero runtime dependencies, plus a small Python generator (unicode.py) used at maintenance time to regenerate the property tables. A no_std Cargo feature swaps std for equivalent core functionality.

Code Quality - The code is short, stable, and heavily relied upon, with a dedicated tests.rs exercising the identifier checks. Because the data is generated rather than hand-written, correctness follows directly from the Unicode source data, and the crate has been battle-tested through years of use in the Rust toolchain and macro ecosystem.

API Design - The public surface is about as small as an API can be: import the UnicodeXID trait and call two boolean methods directly on any char. There is essentially no boilerplate or configuration, which is exactly what a lexer inner loop needs.