How to use the Indexability Conflicts Auditor agent skill

What an agent skill is and what this one makes the agent do

An agent skill is a folder of instructions that an AI agent loads when a task matches the skill's description. This one covers indexability: the rules that decide whether search engines crawl a page and whether it appears in the index.

Once installed, the agent knows the rules for spotting SEO signals that conflict. When asked to audit a site, it checks four signals for each page: whether the URL path is blocked by robots.txt, whether the HTML has a meta robots noindex tag, whether the HTTP response has an X-Robots-Tag noindex header, and whether the canonical link points to a different URL. It then reports the conflicts it finds and suggests fixes, such as removing the robots.txt block or pointing canonicals at indexable URLs.

Install the skill from the ZIP or the Claude apps

Download the ZIP from this page and unzip it. For a personal setup, place the unzipped skill folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder.

If you use the Claude apps instead, upload the skill under Settings → Capabilities → Skills.

• Personal install: ~/.claude/skills/
• Project install: the project's .claude/skills/ folder
• Claude apps: Settings → Capabilities → Skills

Trigger the skill with a natural request

The agent decides to use the skill from the skill's description, so you do not need a special command. Ask in plain terms about auditing indexing, why pages show up in Google when they should not, or whether robots.txt and meta robots settings agree.

Give the agent the site or the list of URLs you care about. The more specific the request, the more useful the report.

• Audit the indexing settings on example.com and report any conflicts between robots.txt and meta robots.
• Some of our old pages show in Google results. Check whether they carry noindex signals that contradict robots.txt.
• Review our robots.txt and meta robots setup before the site migration goes live.

What the audit checks and how fixes are chosen

The skill focuses on mismatches that search engines cannot resolve. The most serious is a page blocked in robots.txt that also has noindex: crawlers never fetch the page, so the noindex is never read, yet the URL can still be known from sitemaps or links.

For each conflict the skill gives a fix path. If you want the page out of the index, remove the robots.txt rule and keep noindex so crawlers can read it. If you want to block all crawling, keep the robots.txt block and remove noindex. Canonicals that point to a noindex page should be changed to an indexable URL. Where index and noindex both appear in meta robots, Google uses the most restrictive directive, so pick one clear intent.

The skill also recommends verifying each fixed page with Google Search Console URL Inspection.

• robots.txt blocks crawling; noindex blocks indexing. They are different mechanisms.
• Canonical targets must be indexable pages.
• Verify fixes in Google Search Console URL Inspection.

Run an audit step by step

Work through the site in batches. For each URL, the agent should compare the robots.txt match against the page's meta robots tag and X-Robots-Tag header, then check the canonical target. Ask the agent to list each flagged URL with its conflict type so you can review them one by one.

The skill's own code review notes describe the approach: fetch robots.txt, parse its Disallow rules, match each page URL, then fetch the HTML and headers for URLs that match a Disallow pattern. The skill does not ship a finished script, so the agent may need to write one for you.

Safety notes: scripts and network access

This skill does not run scripts and does not use the network by itself. Any fetching of robots.txt, pages or headers would be code the agent writes during the task, so review that code before running it against a live site.

Editing robots.txt or canonical tags incorrectly can deindex important pages. Test changes on a staging copy where possible and check results after each change.

• Runs scripts: no
• Uses network: no
• Required tools: none listed
• Review any agent-written fetch code before running it on production

Tips for better results

Give the agent the exact URLs or sections of the site to check, and say whether you want pages indexed or excluded. The skill handles intent conflicts better when the goal is stated.

The guidance follows Google's documentation as summarised in the skill files. Check current Google documentation before acting, because search engine behaviour can change.

• State the intended outcome for each page type.
• Ask for the conflict type on every flagged URL.
• Re-run the audit after fixes and confirm in Google Search Console.