CrawlerDetect

CrawlerDetect is a PHP library that identifies bots, crawlers, and spiders by matching the User-Agent and HTTP_FROM headers against a regularly updated database of thousands of known crawler signatures. It ships as a single Composer package with a minimal API surface: instantiate the class and call isCrawler() to get a boolean answer, or getMatches() to see which pattern matched.

The project has been maintained since 2015, sees active commits and frequent releases, and has been ported to several other frameworks and languages (Laravel, Symfony, Yii2, and more), reflecting broad real-world adoption. Its crawler signature data is contributed and curated by the community, with clear guidelines for adding new bot patterns via pull request.

What You Get

• A CrawlerDetect class with isCrawler() for a straightforward true/false bot check
• getMatches() to retrieve which specific crawler signature matched
• A regularly updated fixture of thousands of known crawler/bot regex patterns
• Support for checking either the current request’s user agent or an arbitrary string you pass in
• Official ports and integrations for Laravel, Symfony, Yii2, and other frameworks

Common Use Cases

• Excluding bot traffic from analytics or A/B testing pipelines
• Serving lightweight, crawler-optimized responses (e.g., pre-rendered HTML) to known bots
• Blocking or rate-limiting abusive scrapers at the application layer
• Filtering bot sessions out of user-facing features like ‘who’s online’ widgets

Under The Hood

Architecture - The core CrawlerDetect class in src/ loads a large static array of crawler regex patterns from src/Fixtures/Crawlers.php and compiles them into a single combined regular expression for efficient matching against a user-agent string. This fixture-driven design keeps the detection logic itself tiny while the pattern data — the actual source of the library’s value — is contributed and reviewed separately by the community via pull requests, with raw/Crawlers.json and raw/Crawlers.txt regenerated automatically after merge.

Tech Stack - It’s a pure-PHP, dependency-free Composer package (jaybizzle/crawler-detect) with PSR-4 autoloading, requiring no external services or runtime libraries. Tests run via PHPUnit with GitHub Actions CI and Coveralls coverage reporting.

Code Quality - The tests/ directory includes a large corpus of real-world user-agent strings (tests/data/user_agent/crawlers.txt) used to regression-test every known pattern, which is an effective strategy for a signature-matching library where correctness hinges on exhaustive test data rather than complex logic. Active CI, 100+ tagged releases, and ~990 commits from 110 contributors indicate a healthy, well-reviewed contribution process.

API Design - The public API is deliberately tiny: construct new CrawlerDetect(), call isCrawler() (optionally passing a user-agent string), and optionally call getMatches() for the matched pattern name. There’s effectively no configuration or boilerplate required to get a working bot check.