Skip to content
Get packs

Guide · Agent skill

How to use the AI Crawler Access Checker agent skill (geo-crawlers)

Updated · written with AI from the project's own documentation and checked against our catalogue data

The geo-crawlers skill is an agent skill for Claude Code and other SKILL.md-compatible agents that audits a website to see which AI company crawlers can reach it. It checks robots.txt rules, meta tags and HTTP headers, then returns an access map with recommendations. This guide explains what the skill does, how to install it, how to ask for an audit in plain language, and what to check before you act on the results.

AI Crawler Access Checkerzubair-trabzada/geo-seo-claude · MIT

Quick start

  1. 1Download the ZIP. Download the geo-crawlers ZIP from this page and unzip it so SKILL.md is available.
  2. 2Place the folder. Move the geo-crawlers folder into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder.
  3. 3Confirm the agent can load it. Check that your agent has the required tools and internet access, or upload the skill under Settings → Capabilities → Skills in the Claude apps.
  4. 4Ask for an audit. Name the site and ask in plain language, for example asking which AI crawlers can access it.
  5. 5Review the access map. Read which crawlers are allowed or blocked and why, based on robots.txt, meta tags and HTTP headers.
  6. 6Decide what to change. Apply only the changes you agree with, after reviewing the Write access and keeping a backup of your current configuration.

What is an agent skill, and what does geo-crawlers do?

An agent skill is a folder containing a SKILL.md file that gives an AI agent specific instructions and tools for one kind of task. When the skill is installed, the agent can load these instructions when your request matches the skill's description.

The geo-crawlers skill is aimed at AI crawler access analysis. AI companies use bots such as GPTBot, ClaudeBot and PerplexityBot to discover and index web content. If those bots are blocked, the source says your content cannot appear in AI-generated responses, regardless of its quality. The skill checks which of these crawlers are allowed or blocked and produces an access map with recommendations.

  • Checks robots.txt rules, meta tags and HTTP headers
  • Builds an access map of AI crawlers for the site
  • Includes a reference list of crawlers, their operators and their purposes
  • Gives recommendations, which generally favour allowing the crawlers that power AI search

Install the geo-crawlers skill

Download the ZIP for skills/geo-crawlers from this page and unzip it. The skill's only listed file is SKILL.md, so the unzipped folder should contain that file.

For a personal install, place the geo-crawlers folder in ~/.claude/skills/. For a single project, place it in the project's .claude/skills/ folder instead.

Alternatively, upload the skill in the Claude apps under Settings → Capabilities → Skills.

  • Personal: ~/.claude/skills/geo-crawlers/
  • Project only: .claude/skills/geo-crawlers/ inside your project
  • Claude apps: Settings → Capabilities → Skills, then upload
~/.claude/skills/geo-crawlers/SKILL.md

Check what the skill needs before you run it

The skill declares the tools it may use: Read, Grep, Glob, Bash, WebFetch and Write. It needs Claude Code or another SKILL.md-compatible agent that provides these tools, plus internet access so it can fetch the target site's robots.txt and headers.

The skill runs scripts and uses the network, as listed on this page. Only install it if you are comfortable with an agent running commands and fetching pages on your behalf.

  • Agent: Claude Code or another SKILL.md-compatible agent
  • Tools: Read, Grep, Glob, Bash, WebFetch, Write
  • Internet access to the site being audited

Trigger the skill with a natural request

You do not need a special command. Describe what you want in plain language that matches the skill's purpose: checking AI crawler access, auditing robots.txt for AI bots, or asking whether ChatGPT or Claude can reach a site. The agent should load the skill when the request fits its description.

Name the site you want audited. Ask for the access map if you want a per-crawler breakdown, or for recommendations if you want to know what to change.

  • "Check which AI crawlers can access example.com."
  • "Does my robots.txt accidentally block GPTBot or ClaudeBot?"
  • "Run a GEO crawler access audit on this site and give me an access map."
  • "Which AI bots are blocked by meta tags or HTTP headers on this page?"

Read the access map and recommendations carefully

The output is an access map showing which crawlers are allowed or blocked and why, based on robots.txt rules, meta tags and HTTP headers. The recommendations generally advise allowing the crawlers that power AI search, such as OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. Google-Extended, which controls Gemini training and AI Overviews improvement, is also recommended to be allowed.

The source notes that blocking Google-Extended does not affect Google Search rankings, which are controlled by the standard Googlebot. It also distinguishes training-focused crawlers, such as GPTBot, from search-only ones like OAI-SearchBot.

  • Tier 1 (AI search): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, all recommended ALLOW
  • Tier 2 (wider AI ecosystem): Google-Extended, GoogleOther, Applebot-Extended, Amazonbot, FacebookBot, all recommended ALLOW
  • Tier 3 (training-only crawlers): the source says these are ALLOW or BLOCK depending on strategy

Safety notes: review the recommendations and the write access

The skill analyses and advises. It does not change a live site automatically. The reviewed SKILL.md lists Write access, so check what the agent writes before you let it make changes, and never apply robots.txt edits to production without reading them.

The recommendations lean strongly toward allowing crawlers. If your goal is to block AI training on your content, do not accept the defaults. Review each crawler choice yourself.

  • Review every recommendation before editing robots.txt, meta tags or headers
  • Keep a copy of your current robots.txt before any change
  • Be aware that Write is in the allowed tools list

What people use it for

  • A site owner checks whether an older robots.txt written for search engines also blocks ChatGPT, Claude or Perplexity.
  • A developer audits a staging or production site to confirm that GPTBot and ClaudeBot can reach key pages.
  • A marketer runs a generative engine optimisation (GEO) audit and wants a per-crawler access map to share with the team.
  • An SEO team compares meta tags and HTTP headers across the site to find unexpected AI crawler restrictions.

Frequently asked questions

How do I install the geo-crawlers skill?

Download the ZIP from this page and unzip it. Put the geo-crawlers folder in ~/.claude/skills/ for personal use or in a project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.

What do I need to run geo-crawlers?

You need Claude Code or another SKILL.md-compatible agent with Read, Grep, Glob, Bash, WebFetch and Write tools, plus internet access to fetch the target site's robots.txt and headers. The skill runs scripts and uses the network.

Is the geo-crawlers skill free to use?

The source lists the skill under the MIT licence. Check the licence file in the repository for the full terms before you reuse or redistribute it.

Will geo-crawlers block AI bots on my site automatically?

No. The skill analyses your site and gives recommendations. It does not change a live site automatically, but its tools include Write access, so review any change before applying it.

Can I use geo-crawlers if I want to block AI training?

You can, but the skill's recommendations lean strongly toward allowing crawlers. Review each crawler choice yourself, especially training-focused bots such as GPTBot and Google-Extended, before making changes.

What is the difference between GPTBot and OAI-SearchBot?

According to the skill's reference, GPTBot fetches content for ChatGPT browsing and features and may be used to improve OpenAI models. OAI-SearchBot powers ChatGPT search, and content it accesses is not used for model training.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use geo-crawlers if you run a site or SEO audit and want to know which AI crawlers can reach it, especially if an old robots.txt may be blocking them. Pick something else if your priority is blocking AI training, or if you want an agent that edits your live site without human review.

Download the skill