Scraper

Scraper is a Rust library for parsing HTML documents and fragments and then querying them with familiar CSS selector syntax. It’s built directly on top of Servo’s browser-grade html5ever HTML5 parser and selectors CSS selector-matching engine, so parsing behaves the way a real browser would, including handling malformed markup gracefully.

Once a document is parsed into a Html value, elements are queried with Selector::parse("...") and .select(&selector), returning an iterator of matched elements whose attributes, text, and inner/outer HTML can all be inspected. The underlying tree can also be mutated via the HtmlTreeSink / TreeSink interface for removing or altering nodes, and an optional atomic feature makes the document’s reference-counted strings Send for multi-threaded scraping pipelines.

What You Get

• Document and fragment parsing via Html::parse_document / Html::parse_fragment, powered by Servo’s html5ever HTML5 parser
• CSS selector queries through Selector::parse and .select(), using the same selectors crate that powers Servo/Firefox’s style engine
• Element inspection helpers: .value().attr(), .text(), .html(), and .inner_html() for pulling attributes and text/markup out of matches
• DOM mutation support via the HtmlTreeSink/TreeSink interface for removing or editing nodes after parsing
• An optional atomic feature that swaps in atomically reference-counted Tendril strings so parsed documents can be sent across threads
• Optional serde support for serializing selector/element data

Common Use Cases

• Web scraping pipelines that fetch HTML over HTTP and need to extract structured data with CSS selectors
• Static site or content-migration tooling that parses legacy HTML pages and rewrites or extracts specific elements
• Test suites and crawlers that assert on rendered HTML structure without spinning up a real browser
• Multi-threaded scraping workers that parse many documents concurrently using the atomic feature for thread-safe sharing

Under The Hood

Architecture - scraper is a thin, ergonomic layer over two lower-level Servo crates: html5ever does the actual HTML5 tokenizing/tree-construction and selectors does CSS selector matching against the resulting tree; the crate’s own code (src/html/, src/selector.rs, src/element_ref/, src/node.rs) mostly adapts those engines’ tree and selector types into an ergonomic Html/Selector/ElementRef API, with src/error.rs and src/main.rs providing error types and an optional CLI binary respectively. Tech Stack - Rust 2024 edition, depending on html5ever 0.39, selectors 0.39, cssparser 0.37, and ego-tree for the underlying arena-based DOM tree, with indexmap, serde, and getopts as optional features; the main feature builds a small CLI binary alongside the library. Code Quality - The crate ships src/test.rs plus scattered #[test] blocks across roughly 8 files covering selector parsing and DOM querying edge cases, and it has shipped 19 tagged releases with steady minor-version bumps as html5ever/selectors are upgraded, indicating an established regression-testing habit. API Design - The API mirrors familiar browser DOM/CSS-selector semantics almost one-to-one (Selector::parse("h1.foo"), .select(), .text(), .html()), so anyone who has used document.querySelectorAll in JavaScript can be productive within minutes, and the README’s example-driven documentation covers parsing, selecting, attribute access, and serialization with minimal boilerplate per example.