Texting Robots

Texting Robots is a Rust library for parsing robots.txt files and deciding whether a given URL is crawlable for your bot. It exposes a single high-level Robot struct that consumes a robots.txt payload, resolves the applicable user-agent rules, and answers allow/disallow queries, while also surfacing crawl-delay and sitemap directives.

Designed with correctness as the primary goal, the crate ports almost all unit tests from Google’s C++ robotstxt parser and Moz’s reppy, and its parser was hardened by running over 34 million real robots.txt responses from Common Crawl to ensure it never panics on adversarial or malformed input.

What You Get

• A high-level Robot struct that parses robots.txt and answers allow/disallow URL checks
• Access to Crawl-Delay values and declared Sitemap URLs for the selected user agent
• A get_robots_url helper that derives the correct robots.txt location from any page URL
• A parser validated against Google’s and Moz’s test suites plus 34M+ Common Crawl files

Common Use Cases

• Building a polite web crawler or scraper that respects site owners’ rules
• Checking whether a specific path is permitted for a named bot before fetching it
• Extracting sitemap URLs and crawl-delay hints while planning a crawl schedule

Under The Hood

Architecture - The crate centers on a Robot struct (src/lib.rs, ~559 lines) that is constructed from a user-agent string and a robots.txt byte slice. Construction delegates to a nom-based line parser in src/parser.rs that lexes the file into a typed Line enum (UserAgent, Allow, Disallow, Sitemap, CrawlDelay, Raw); matching rules for the requested agent are compiled into a minregex matcher (src/minregex.rs) that evaluates path patterns for allow/disallow decisions, exposing allowed, delay, and sitemaps.

Tech Stack - Pure Rust (edition 2021) built on nom 7 for parsing, bstr for byte-string handling, regex and a lightweight minregex wrapper for path matching, plus url/percent-encoding for URL normalization and anyhow/thiserror for errors. Release builds enable LTO.

Code Quality - Testing is exceptional relative to the small surface area: over 1,200 lines of unit tests in src/test.rs (57+ test functions), an integration suite in tests/, ported Google/Moz conformance tests, and a fuzz harness under fuzz/. Naming is clear, errors are typed via thiserror, and Tarpaulin coverage is tracked.

API Design - The public API is intentionally minimal — mostly Robot::new, allowed, delay, sitemaps, and get_robots_url — so getting started takes only a few lines, as the README example shows. Leaving HTTP fetching to the caller keeps the ergonomics simple and the dependency footprint light.