Skip to content
Get packs

AI Crawler Access Checker

Checks whether AI bots like GPTBot and ClaudeBot can reach your site, and recommends robots.txt changes.

Source Download skill
Repository
zubair-trabzada/geo-seo-claude
Stars
11,419
Licence
MIT
Files
1 file

What it does

When installed, the agent can audit a website to see which AI company crawlers (bots such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended) are allowed or blocked by robots.txt rules, meta tags and HTTP headers. It then produces an access map and recommendations, generally advising the site owner to allow the crawlers that power AI search so their content can appear in AI-generated answers. The file contains a reference list of these crawlers and their purposes, with the skill's guidance on each.

Great for

  • Site owners or SEO teams who want their content visible in AI search tools like ChatGPT, Claude and Perplexity
  • Developers checking whether an old robots.txt accidentally blocks AI crawlers
  • Marketers doing a generative engine optimisation (GEO) audit

Skip it if

  • Sites whose owners want to block AI training on their content: the skill's recommendations lean strongly toward allowing crawlers, so review each choice yourself
  • Anyone expecting it to change a live site automatically: it analyses and advises, and the reviewed SKILL.md grants Write access that should be checked before use

Try asking

  • “Check which AI crawlers can access example.com and list any that are blocked”
  • “Review my robots.txt and tell me whether GPTBot or ClaudeBot is being blocked”
  • “Give me a GEO crawler access report with recommendations for my homepage”
Step-by-step guide: how to use AI Crawler Access CheckerInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 1 original files, the MIT licence and an attribution note.

    geo-crawlers.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip geo-crawlers.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip geo-crawlers.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Run it on a staging copy or ask it to report only, so Write is not used to change your live robots.txt without review
  • Weigh the recommendations against your own policy on AI training before changing blocks
  • Check the rest of the repository's scripts, if any, because the skill's allowed tools include Bash

What's inside

  • SKILL.md

Redistributed unmodified from zubair-trabzada/geo-seo-claude under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.