Guide · Agent skill
How to use the BrightData Page Fetcher agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The BrightData skill lets an AI agent pull the content of a web page, or crawl a whole site, and return it as markdown text. It is for developers, researchers and analysts who need page text for documentation, research or LLM input. After reading this guide you will know what the skill does, how to install it, how to ask for it in plain language, and what extra setup the paid options need.
BrightData Page Fetcherdanielmiessler/LifeOS · MIT Download the skill Quick start
- 1Download the ZIP. Download the BrightData skill ZIP from this page.
- 2Unzip into a skills folder. Unzip it into ~/.claude/skills/ for all projects, or into a project's .claude/skills/ folder.
- 3Install Bun. Install the Bun runtime, which runs the skill's TypeScript helper scripts.
- 4Ask for a page. Request a URL in plain language, such as fetch this page as markdown.
- 5Check the reported tier. Read the tier label on the result to see which method retrieved the page.
- 6Set up paid pieces only if needed. Register the Bright Data MCP server and add credentials to ~/.claude/.env if you want Tier 4 or Full Crawl.
What the BrightData skill makes the agent do
The skill fetches a single URL or crawls a site by following links. It tries four tiers in order and stops at the first one that works: a plain curl request, a curl request with declared browser-like headers, a real Chrome browser through the Interceptor skill, and finally Bright Data's paid managed fetch product.
Every page comes back as markdown labeled with the tier that retrieved it. If a site refuses automated access, a CAPTCHA appears, or a login wall blocks the page, the skill reports that and stops rather than working around it.
- Tier 1: plain curl, raw HTML converted to markdown locally. Free.
- Tier 2: curl with declared user agent and Accept headers. Free.
- Tier 3: Interceptor (real Chrome) for JavaScript-rendered pages. Free.
- Tier 4: Bright Data MCP tool scrape_as_markdown for managed rendering and region-specific fetches. Uses Bright Data credits.
Install the skill (ZIP download or Claude apps upload)
Download the BrightData ZIP from this page. Then choose one of two locations. For a personal install available in all projects, unzip it into ~/.claude/skills/. For a single project, unzip it into that project's .claude/skills/ folder.
If you use the Claude apps instead, open Settings, go to Capabilities, and upload the skill under Skills.
- Personal: ~/.claude/skills/
- Project only: .claude/skills/ inside your project
- Claude apps: Settings → Capabilities → Skills
What you need before it runs
The helper scripts are TypeScript and run with the Bun runtime, so Bun must be installed. The free tiers need curl and the Interceptor skill for Tier 3.
The paid paths need more. Tier 4 needs the Bright Data MCP server registered in your agent. Full Crawl additionally needs BRIGHT_DATA_API_KEY and BRIGHT_DATA_CRAWL_DATASET_ID in ~/.claude/.env, and a Bright Data account with credits. The skill's Setup.md file walks through these pieces and includes a check that they work.
- Bun runtime (runs the TypeScript helper scripts)
- Bright Data MCP server, for Tier 4 and paid paths only
- BRIGHT_DATA_API_KEY and BRIGHT_DATA_CRAWL_DATASET_ID in ~/.claude/.env, for Full Crawl only
- A Bright Data account with credits, for paid tiers only
Trigger the skill with natural requests
You do not need a special command. Ask in plain language and the agent matches the request. Phrases such as scrape, fetch, pull, get or retrieve with a URL route to single-page fetching. Phrases such as crawl this site, spider this domain or get all pages from route to crawling.
To skip ahead, name the tier. Saying use Bright Data goes straight to Tier 4, and saying use browser goes to Tier 3.
- Fetch example.com/pricing as markdown
- Crawl the docs under docs.example.com/api/
- Use Bright Data to fetch this region-specific listing
- This page needs rendering, fetch it as markdown
Examples of what you get back
For a plain request such as fetching a pricing page, the pre-check may return HTML and Tier 1 may return an empty body. Tier 2's declared headers can then retrieve the page, and the reply is delivered as markdown labeled Tier 2.
For a crawl of a documentation section, the skill runs a Light Crawl scoped to the path, with depth 3 and a default limit of 20 pages, which needs no confirmation. Each page climbs the tiers on its own. The stats name any page that needed Tier 4 and its cost. A crawl of more than 20 pages needs your confirmation.
Safety notes: scripts and network access
This skill runs scripts and uses the network. Its fetching stays within each target site's terms and robots.txt. Scraping private data may also breach site terms, so check before you fetch content behind a login or about individuals.
The skill's Tier 4 and Full Crawl use Bright Data, a paid service. The skill is designed to avoid paid calls when a free tier succeeds, but you should keep an eye on credit usage.
- Runs scripts: yes
- Uses network: yes
- Does not handle social media data via named data-provider actors; that is excluded
- Does not handle logged-in or private browser sessions; those belong to the Interceptor skill
Tips for better results
Name the URL exactly and say whether you want a single page or a crawl. If you say crawl without a scope, the skill asks how many pages or which section you want first.
If a paid tier is not configured, the skill stops at the last free tier and tells you which piece is missing. Set up the MCP server and credentials once, then the paid paths work without changes to your requests.
- Give the full URL and a path scope for crawls
- Use Bright Data only when a free tier has failed or you need region-specific content
- Check the Setup.md file if the paid tier reports a missing piece
What people use it for
- Pull a documentation page into markdown for a local reference or LLM input.
- Crawl a documentation section such as /api/ into a site map with page content.
- Collect public pages from a known list of URLs for analysis.
- Fetch a JavaScript-rendered page through the Interceptor browser tier.
- Retrieve a region-specific listing through Bright Data's managed fetch.
Frequently asked questions
Is the BrightData skill free?
Tiers 1 to 3 are free and need no account, according to the skill's Setup.md. Tier 4 and Full Crawl use Bright Data credits, so they need a Bright Data account. The skill's source text does not give prices beyond a note that Full Crawl has a cost estimate of $1.50 per 1K pages.
What do I need to install the BrightData skill?
You need the Bun runtime to run the TypeScript scripts, and curl for the free tiers. Tier 3 needs the Interceptor skill. Paid paths also need the Bright Data MCP server and, for Full Crawl, a Bright Data API key and crawl dataset ID in ~/.claude/.env.
Where do I put the BrightData skill files?
Unzip the downloaded ZIP into ~/.claude/skills/ to use it in all projects, or into .claude/skills/ inside one project. In the Claude apps, you can instead upload it under Settings, then Capabilities, then Skills.
How is this different from WebFetch?
The skill returns the page itself as markdown. WebFetch returns a small model's summary of the page. For simple public pages, the skill's own documentation says WebFetch is enough and this skill adds setup steps.
Why does the skill say a page is blocked?
A CAPTCHA, block page or login wall at any tier ends the retrieval with a report. The skill is designed not to bypass sites that refuse automated access, so it reports the refusal instead.
Can it scrape social media or logged-in pages?
No for these cases. The skill excludes social data from named data-provider actors, and logged-in or private browser sessions belong to the Interceptor skill.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use the BrightData skill if you are a developer or analyst who regularly needs public page content or whole documentation sites in markdown, and you want free methods tried before paid ones. Pick WebFetch for a single simple public page, and pick Interceptor for logged-in browser work.
Download the skill