html5gum

html5gum is a Rust crate that implements the WHATWG HTML5 tokenization algorithm, letting applications turn raw (and often malformed) HTML “tag soup” into a stream of tokens without pulling in a full browser-grade parser. It ships as #![forbid(unsafe_code)] and, by default, depends only on the small jetscii crate for fast byte scanning — a build with the default feature disabled has zero dependencies at all.

What sets html5gum apart is its layered emitter API: consume tokens as a plain iterator for convenience, plug in a callback-based emitter to avoid most allocations, implement the Emitter trait yourself for full control, or enable the tree-builder feature to hand tokens off to html5ever’s DOM builder and use it alongside scraper. It deliberately stops at tokenization — no charset detection, no mis-nested tag correction, no DOM — which keeps the core small while still passing the full html5lib conformance test suite.

What You Get

• An iterator-based Tokenizer that yields Result<Token, Error> from any &str/String input with three lines of code
• A pluggable Emitter trait with three ready-made strategies: DefaultEmitter for convenience, CallbackEmitter for a low-allocation SAX-like API, or a fully custom implementation
• Optional tree-builder feature integrating with html5ever’s DOM construction and the scraper crate for CSS-selector querying
• Span tracking (Span/Spanned) so tokens can be mapped back to byte offsets in the original input
• A safety guarantee (#![forbid(unsafe_code)]) suited to parsing untrusted HTML from the internet
• An optional zero-dependency build by disabling the default jetscii feature

Common Use Cases

• Extracting text, links, or metadata from arbitrary web pages without pulling in a browser engine
• Building web scrapers or crawlers that need to tolerate malformed HTML gracefully
• Implementing custom HTML sanitizers or rewriters via a hand-written Emitter
• Feeding tokens into html5ever’s tree builder to get a full DOM plus CSS-selector querying via scraper
• High-throughput log or document pipelines that need HTML tokenization without per-token heap allocation

Under The Hood

Architecture The Tokenizer<R, E> struct is the center of the crate: it owns a CharValidator, a generic Emitter, a ReadHelper<R> wrapping the input Reader, and a MachineHelper<R, E> holding the current state’s function pointer. Iterator::next first drains any buffered token from emitter.pop_token(), and otherwise calls the active state function, which returns a ControlToken (Continue, SwitchTo(next_state), or Eof) driving the machine forward — a classic finite-state-machine interpreter, with the states themselves defined in a dedicated state-machine module the project’s own README half-jokingly calls a “giant unreadable match-statement.” The Emitter trait is a visitor that decouples state transitions from token construction, so DefaultEmitter, CallbackEmitter, and the optional html5ever-backed emitter can all drive the same core loop; the Reader/Readable traits similarly abstract the input source. This separation means the tokenizer core stays untouched when adding new emitters or input sources, though changes to the Emitter trait itself ripple through every implementation.

Tech Stack Built against the 2018 Rust edition with no required runtime dependencies — the default jetscii feature adds fast byte-pattern scanning and can be turned off for a fully dependency-free build. The optional tree-builder feature pulls in html5ever for DOM construction, pairing naturally with the scraper crate shown in the example set. Fuzzing support (afl) is feature-gated separately. The dev-dependency set is oriented entirely around spec conformance and benchmarking: test-generator and libtest-mimic drive custom test harnesses over the html5lib-tests fixture suite (pulled in as a git submodule), markup5ever_rcdom backs the tree-builder tests, and an iai-based cachegrind benchmark suite reports to bencher.dev in CI. Distribution is standard crates.io plus docs.rs.

Code Quality Testing is unusually rigorous for a crate of this size: rather than ad hoc unit tests, the primary correctness suite runs the canonical WHATWG html5lib-tests JSON fixtures through custom libtest-mimic harnesses (tests/html5lib_tokenizer.rs, tests/html5lib_tree_builder.rs), supplemented by a dedicated span-tracking test file and a project-local set of custom-case fixtures. #![forbid(unsafe_code)] is enforced crate-wide, and lib.rs layers on an extensive lint list (missing_docs, unreachable_pub, missing_debug_implementations, variant_size_differences, and clippy::all denied in CI). Errors are modeled as an explicit Error enum threaded through Result rather than panics. CI additionally compiles AFL and libFuzzer fuzz targets on every push, appropriate given the crate’s own stated threat model of parsing untrusted HTML from the internet.

API Design The entry point is deliberately minimal — Tokenizer::new(input) returns a plain iterator of tokens, usable in three lines with no setup. From there the API graduates in a documented ladder: swap in CallbackEmitter to avoid most token allocation, or implement Emitter directly for maximum control, with the module docs explicitly framing this as a convenience-versus-performance tradeoff. The tree-builder feature smooths interop with the broader Rust HTML ecosystem (html5ever, scraper) instead of forcing consumers to write their own tree builder. The tradeoff for this flexibility is that the generic Reader/Emitter trait bounds add real complexity for anyone implementing a custom emitter, and the crate is explicit in its README about what it intentionally does not do — charset detection, mis-nested tag correction, and full DOM construction are all left to other crates.