syntect

syntect is a Rust library for syntax highlighting and basic code intelligence, built on top of Sublime Text’s .sublime-syntax grammar format and .tmTheme color schemes. Because it reuses the same battle-tested grammars Sublime Text ships with, it gets near-complete language coverage and correctness for free, including tricky edge cases like heredocs and deeply nested embedded languages.

The library exposes both a simple one-call API for common cases (highlight a string, get styled HTML or ANSI-escaped terminal output) and lower-level primitives — parse states, scope stacks, cacheable parser snapshots — that let text editors implement incremental re-highlighting as a user types, without re-parsing the whole file on every keystroke.

What You Get

• A bundled, compressed dump of Sublime Text’s default syntax and theme definitions, ready to use out of the box
• An easy API (HighlightLines) for one-shot highlighting of a string into styled spans
• HTML output helpers (html module) for rendering highlighted code as <pre> blocks
• ANSI 24-bit terminal escape output for CLI tools
• Exposed internal parse state so editors can cache and incrementally re-highlight only the changed region of a file
• A choice of regex engines — the default onig (Oniguruma) binding or a pure-Rust fancy-regex mode with no C dependency

Common Use Cases

• Rendering syntax-highlighted code blocks in static site generators and documentation tools
• Building terminal-based file viewers or cat-style tools with colorized code output
• Implementing incremental re-highlighting in Rust-based text editors and IDEs
• Generating highlighted HTML for code snippets in web applications and blogging platforms

Under The Hood

Architecture - syntect splits cleanly into a parsing module (src/parsing/) that interprets Sublime .sublime-syntax grammars into a scope-annotated token stream, and a highlighting module (src/highlighting/) that maps those scopes to a .tmTheme color theme; easy.rs composes the two into a single-call API, while html.rs and util.rs provide renderer helpers for HTML and ANSI terminal output respectively. dumps.rs handles serializing/deserializing pre-compiled syntax sets via bincode+flate2 so consumers don’t pay the cost of parsing every grammar file at startup.

Tech Stack - Rust 2021 edition, using onig (Oniguruma C bindings) or fancy-regex (pure Rust) as pluggable regex backends selected via Cargo features, serde/serde_derive for grammar/theme deserialization, yaml-rust2 for parsing YAML-based syntax files, thiserror for error types, and walkdir for scanning syntax directories. Feature flags let consumers opt out of parsing or highlighting independently to shrink dependency footprint.

Code Quality - The repo carries a substantial tests/ suite including snapshot tests (tests/snapshots), a dedicated error_handling.rs test file, and a public_api.rs test guarding against accidental breaking API changes via rustdoc-json/public-api. Combined with criterion benchmarks and years of production use (cited by at least two companies and many OSS editors), the codebase shows mature test discipline for a parsing-heavy library.

API Design - The library favors a layered API: easy::HighlightLines covers the 90% case in a few lines, while lower-level parsing::ParseState/highlighting::Highlighter primitives are exposed for advanced use like incremental re-highlighting caches — a deliberate trade-off that keeps the common path simple without hiding the internals editors need for performance-sensitive integrations.