Guide · Open source

How to use Scrapegraph-ai: install and run your first scraper

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

Scrapegraph-ai (ScrapeGraphAI) is an MIT-licensed Python library that uses a large language model and graph-based pipelines to pull information from websites or local files such as HTML, XML, JSON and Markdown. You describe the data you want in a sentence and the library runs the scraping steps and returns the result. This guide is for developers and data people comfortable with Python; by the end you will install the library, run a SmartScraperGraph and understand which LLM options and pipelines are available.

ScrapeGraphAIScrapeGraphAI/Scrapegraph-ai · ★ 32.4k · MIT Install options & downloads

Quick start

1Create a virtual environment. Set up an isolated Python environment so the library does not conflict with other packages.
2Install the library. Run pip install scrapegraphai from PyPI.
3Install browser binaries. Run playwright install so the library can fetch website content.
4Choose your LLM. Pick a hosted provider with an API key, or install Ollama and pull a local model with ollama pull.
5Write the config. Set the llm model and options such as verbose and headless in a graph_config dictionary.
6Run SmartScraperGraph. Pass your prompt, a source URL and the config to SmartScraperGraph, then call run() and print the dictionary.

Install ScrapeGraphAI and Playwright browser dependencies

The library is installed from PyPI with pip. The README recommends using a virtual environment to avoid conflicts with other libraries.

Fetching website content also needs Playwright browser binaries, which are installed with a separate command.

Prerequisites: Python, Playwright browser binaries, and an LLM (an API key for a provider such as OpenAI, or Ollama for local models).

pip install scrapegraphai

# IMPORTANT (for fetching websites content)
playwright install

Choose an LLM: an API provider or a local model with Ollama

The library depends on an LLM for extraction. You can use hosted APIs such as OpenAI, Groq, Azure or Gemini, or run local models through Ollama. The README says that for local models you need Ollama installed and the model downloaded with the ollama pull command.

Switching providers is a change to the llm section of the config. The README notes that for OpenAI and other models you just need to change the llm config.

Local: set model to an ollama/ name, such as ollama/llama3.2, and keep data on your own machine.
Hosted: add your api_key and pick a model such as openai/gpt-4o-mini.

graph_config = {
    "llm": {
        "api_key": "YOUR_OPENAI_API_KEY",
        "model": "openai/gpt-4o-mini",
    },
    "verbose": True,
    "headless": False,
}

Run your first SmartScraperGraph

SmartScraperGraph is the most common pipeline. It extracts information from a single page given a user prompt and a source URL. You create a config, build the graph with your prompt, source and config, then call run().

The README example uses a local Ollama model with format set to json. The result is a Python dictionary you can print or save.

prompt: the plain-language description of the information you want.
source: the URL of the page (or a local file).
config: the llm settings plus options such as verbose and headless.

from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {
        "model": "ollama/llama3.2",
        "model_tokens": 8192,
        "format": "json",
    },
    "verbose": True,
    "headless": False,
}

smart_scraper_graph = SmartScraperGraph(
    prompt="Extract useful information from the webpage, including a description of what the company does, founders and social media links",
    source="https://scrapegraphai.com/",
    config=graph_config
)

result = smart_scraper_graph.run()

import json
print(json.dumps(result, indent=4))

Pick the right pipeline for your job

Besides SmartScraperGraph, the README lists graphs for several jobs. Each one has a multi version that can call the LLM in parallel.

The README describes SearchGraph as multi-page, working on the top n results of a search engine, and SpeechGraph as producing an audio file. ScriptCreatorGraph generates a Python script.

SmartScraperGraph: single page, prompt plus source.
SearchGraph: top n search results of a search engine.
SpeechGraph: single page, generates an audio file.
ScriptCreatorGraph: single page, generates a Python script.
SmartScraperMultiGraph: multiple pages from one prompt and a list of sources.
ScriptCreatorMultiGraph: generates a script for multiple pages and sources.

Know the limits of the open-source library

The open-source library is self-hosted. The README places the LLM, browser setup, proxies and anti-bot handling on you. Proxies and anti-bot handling are included only in the managed cloud API, which is used through the Python or JS/TS SDKs.

If you need to scale without running your own infrastructure, the README points to the managed API. The docs are at docs.scrapegraphai.com.

Open source: you bring the LLM, configure Playwright, and handle proxies.
Managed API: hosted service with managed LLM, stealth browsing and included proxies.

What people use it for

Pull company descriptions, founder names and social links from a website into a dictionary.
Prototype a scraper for pages whose layout changes often, using a prompt instead of a custom parser.
Extract structured fields from local HTML, XML, JSON or Markdown files.
Run a search-based pipeline that collects data from the top results of a search engine.
Feed scraped data into LLM or agent frameworks such as LangChain or LlamaIndex, which the README lists as integrations.

Frequently asked questions

Is ScrapeGraphAI free to use?

The library is released under the MIT licence, as shown in the project's badge. You still need to pay for or run an LLM, because the library depends on one for extraction; hosted providers charge for API use, while Ollama runs models locally.

Do I need to know Python to use ScrapeGraphAI?

Yes. The README shows usage as Python code and requires you to configure an LLM in that code. Non-programmers are better served by the managed service or other tools.

Do I need an OpenAI API key?

No. The README shows OpenAI as one option. You can instead run local models through Ollama by setting the model to an ollama/ name and pulling the model first.

Why do I need to run playwright install?

The README marks it as important for fetching website content. It installs the Playwright browser binaries that the library uses to load pages.

Does ScrapeGraphAI handle proxies and anti-bot blocking?

Not in the open-source library. The README says proxies and anti-bot handling are your responsibility there, and they are included in the managed cloud API.

How does ScrapeGraphAI compare to Scrapy, Crawl4AI, Firecrawl or BeautifulSoup?

ScrapeGraphAI extracts data from a plain-language prompt using an LLM. Scrapy, BeautifulSoup and similar tools rely on code you write for each site's structure. Crawl4AI and Firecrawl are other projects in the same space, and this source does not compare them in detail, so test the options on your own pages.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use ScrapeGraphAI if you are a Python developer who wants structured data from changing pages with a plain-language prompt, and you can run or pay for an LLM. Pick a managed service, a no-code tool, or a traditional parser such as BeautifulSoup or Scrapy if you need built-in proxies and anti-bot handling, or if you do not want to write code.

Install options & downloads