Skip to content
SkillModel-led4,596 tokens in use

geo-crawlers

Maps which of 14 AI crawlers can reach a site through robots.txt, meta tags and HTTP headers, then recommends the robots.txt rules to allow or block.

11,792 stars on the repository MIT Licensezubair-trabzada/geo-seo-claude · skills/geo-crawlers
curl -fsSL https://raw.githubusercontent.com/zubair-trabzada/geo-seo-claude/main/install.sh | bash
View source

Install command as written in the repository's README

Skill label

measured from its files

What installing geo-crawlers costs

47tokens idle
4,596tokens in use

Idle is what sits in context while the skill is installed. In use is its instructions, loaded when it runs. Counts are approximate, made offline with cl100k_base.

Model dependence Model-led

Script-ledMixedModel-led

Mostly instructions, so how well it works depends on the model that follows them.

This describes the skill's structure, how much of it is scripts. It does not say how well it works.

Touches

shelldeclarednetworkdeclaredwrites filesdeclared

Declared tags come from the skill's frontmatter; detected ones from its scripts, its code blocks and the commands it names in inline code. This is detected, not audited.

How it scores

Quality

How well the skill is written, counted from its own files: its frontmatter, its steps, examples, stated output and constraints, its size and whether every file it ships is one it names.How we score it →
Excellent
  • Frontmatter8/15
  • Structure23/25
  • Examples15/15
  • Output15/15
  • Size15/15
  • Resources10/15

Security

Starts at 100 and loses points for each thing in the skill's files that a reader should look at first.How we score it →
Excellent

No findings.

Repository health

Pre-computed score based on development activity, maintenance, community, maturity, and trend momentum.How we score it →
Good
  • Development activity100
  • Maintenance52
  • Community68
  • Maturity32
  • Momentum40

geo-crawlers checks whether AI bots such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and CCBot are allowed to read a site. It parses robots.txt, then inspects meta robots tags and X-Robots-Tag headers on a sample of pages.

It sorts the crawlers into three tiers, scores crawler access out of 100, and writes GEO-CRAWLER-ACCESS.md with the critical issues and a complete recommended robots.txt block.

What You Get

  • GEO-CRAWLER-ACCESS.md with a status table for 14 crawlers across three tiers
  • An AI crawler access score out of 100
  • A list of critical issues, such as blocked search crawlers
  • A complete recommended robots.txt for AI visibility
  • Findings on meta robots, X-Robots-Tag, JavaScript rendering, llms.txt and sitemap access

Common Use Cases

  • Find out why a site never appears in ChatGPT or Perplexity answers
  • Decide which training-only crawlers to block while keeping search crawlers open
  • Detect a Cloudflare managed robots.txt block the site owner never wrote
  • Check Content-Signal directives for AI usage preferences

What Makes It Unique

It separates search crawlers (recommended allow) from training-only crawlers (a policy choice) and says what blocking each one costs. It also checks for the IETF draft Content-Signal directive and for the managed robots.txt block that Cloudflare can prepend, which never appears in the file on the origin server.

Under The Hood

A prose recipe. The agent fetches the site’s robots.txt with WebFetch, fetches a sample of five to ten pages to read meta robots tags and X-Robots-Tag headers, and probes for llms.txt, ai.txt and the ai-plugin.json file. It refers to the repository’s fetch_page.py script for the Cloudflare check, which sits outside the skill folder. It declares Read, Grep, Glob, Bash, WebFetch and Write tools and writes GEO-CRAWLER-ACCESS.md to the working directory.

Join founders building with open source

Opinionated takes, migration guides, cost-saving tips, and insights from the open source ecosystem.

Subscribe on SubstackJoin 750+ subscribers