geo-crawlers checks whether AI bots such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and CCBot are allowed to read a site. It parses robots.txt, then inspects meta robots tags and X-Robots-Tag headers on a sample of pages.
It sorts the crawlers into three tiers, scores crawler access out of 100, and writes GEO-CRAWLER-ACCESS.md with the critical issues and a complete recommended robots.txt block.
What You Get
- GEO-CRAWLER-ACCESS.md with a status table for 14 crawlers across three tiers
- An AI crawler access score out of 100
- A list of critical issues, such as blocked search crawlers
- A complete recommended robots.txt for AI visibility
- Findings on meta robots, X-Robots-Tag, JavaScript rendering, llms.txt and sitemap access
Common Use Cases
- Find out why a site never appears in ChatGPT or Perplexity answers
- Decide which training-only crawlers to block while keeping search crawlers open
- Detect a Cloudflare managed robots.txt block the site owner never wrote
- Check Content-Signal directives for AI usage preferences
What Makes It Unique
It separates search crawlers (recommended allow) from training-only crawlers (a policy choice) and says what blocking each one costs. It also checks for the IETF draft Content-Signal directive and for the managed robots.txt block that Cloudflare can prepend, which never appears in the file on the origin server.
Under The Hood
A prose recipe. The agent fetches the site’s robots.txt with WebFetch, fetches a sample of five to ten pages to read meta robots tags and X-Robots-Tag headers, and probes for llms.txt, ai.txt and the ai-plugin.json file. It refers to the repository’s fetch_page.py script for the Cloudflare check, which sits outside the skill folder. It declares Read, Grep, Glob, Bash, WebFetch and Write tools and writes GEO-CRAWLER-ACCESS.md to the working directory.