robots-parser

robots-parser is a lightweight, dependency-free Node.js library for parsing and querying robots.txt files. It implements the RFC 9309 Robots Exclusion Protocol specification, including wildcard (*) and end-of-line ($) pattern matching, so crawlers and scrapers can determine which URLs they are permitted to fetch for a given user-agent.

Beyond basic allow/disallow checks, it exposes crawl-delay values, sitemap URLs, and preferred-host directives parsed from the file, plus a strict “explicitly disallowed” mode for user-agents that must not fall back to wildcard rules (as required by crawlers like Google AdsBot). The matching algorithm mirrors Google’s own robotstxt implementation, keeping its behavior consistent with how major search engines interpret robots.txt.

What You Get

• A single robotsParser(url, contents) factory function with no runtime dependencies
• isAllowed() / isDisallowed() checks with wildcard and $ end-of-line pattern matching per RFC 9309
• isExplicitlyDisallowed() for crawlers (like Google AdsBot) that must ignore wildcard fallback
• getCrawlDelay(), getSitemaps(), and getPreferredHost() accessors for the other robots.txt directives
• Bundled TypeScript type definitions (index.d.ts)

Common Use Cases

• Web crawlers and scrapers checking whether a URL is safe to fetch before requesting it
• SEO tools auditing a site’s robots.txt for crawl-delay or sitemap directives
• Search engine bots enforcing per-user-agent disallow rules, including strict AdsBot-style exclusions
• Build tools and static site generators validating that generated pages aren’t accidentally blocked

Under The Hood

Architecture The library is a single-file, constructor-based design (Robots.js, ~490 lines) exported through a thin factory in index.js. Parsing happens in a private parseRobots() function that tokenizes robots.txt lines (stripping comments, splitting on ”:”, trimming) and dispatches on directive keyword to mutate the instance’s internal state (_rules keyed by lowercased user-agent, _sitemaps array, _preferredHost string). Matching logic lives in a dedicated matches() function that tracks the set of currently valid match lengths across a pattern, ported from Google’s C++ robotstxt implementation, so multiple wildcards are handled without backtracking. Public methods (isAllowed, isDisallowed, isExplicitlyDisallowed, getMatchingLineNumber) all funnel through a shared private _getRule() helper, which validates protocol/hostname/port against the parsed base URL before delegating to findRule() to pick the longest, most-specific matching rule.

Tech Stack Plain, untranspiled JavaScript with zero runtime dependencies, built against Node’s global URL API (engines requires Node >=10). Dev dependencies are limited to chai, mocha, and nyc for testing and coverage, plus hand-written TypeScript definitions rather than a TypeScript source. There is no bundler or build step; npm test runs nyc+mocha directly against the CommonJS source, formatting is governed by a Prettier config embedded in package.json, and CI runs on GitHub Actions with a dedicated CodeQL workflow for security scanning.

Code Quality The test suite (test/Robots.js) is extensive, covering wildcard matching, EOL anchors, user-agent normalization, crawl-delay, sitemaps, preferred host, and explicit-disallow behavior, with coverage tracked via Coveralls. Error handling favors returning null/undefined over throwing: parseUrl() wraps URL construction in try/catch, and public methods return undefined rather than raising when a URL doesn’t match the robots.txt’s origin. Naming is consistent throughout (camelCase methods, underscore-prefixed private fields), and while the source itself isn’t authored in TypeScript, the bundled .d.ts definitions give consumers type safety without a separate @types package.

API Design The public surface is deliberately small: one factory function returning an object with seven methods whose names read as plain-English questions matching the vocabulary of the spec itself (allow/disallow/crawl-delay/sitemap/host). The one method whose semantics are easy to misuse, isExplicitlyDisallowed(), is called out explicitly in the README with guidance on when to use it (AdsBot-style crawlers), which is good hygiene for an API surface this compact. Bundled type definitions mean TypeScript consumers get full autocompletion with no extra install step.