Skip to content
Get packs

ScrapeGraphAI

Python library that uses an LLM to scrape websites and local documents based on a plain-language prompt.

GitHub Download v2.3.1
Stars
32.4k
Forks
3.3k
Licence
MIT
Language
Python
Last update
1 months ago
Type
Library

What it is

ScrapeGraphAI is a Python web scraping library that uses large language models and graph-based pipelines to pull information from websites or local files such as HTML, XML, JSON, and Markdown. You describe the information you want in a sentence, and the library runs the scraping steps and returns the result. It is used by people who want structured data from pages without writing a custom parser for each site.

  • Extract a company description, founders, and social links from a single web page using SmartScraperGraph
  • Scrape the top search results for a query with SearchGraph
  • Run one prompt across multiple pages with SmartScraperMultiGraph
  • Generate a Python scraping script from a website with ScriptCreatorGraph
  • Turn a page's extracted content into an audio file with SpeechGraph

Great for

  • Developers who want to extract structured data from web pages using a plain-language prompt
  • Data teams prototyping scrapers for pages with changing layouts
  • People who want to run local models through Ollama and keep data on their own machines
  • Builders connecting scraping to LLM or agent frameworks such as LangChain or LlamaIndex

Not ideal for

  • Non-programmers, since usage requires writing Python code and configuring an LLM
  • Anyone needing managed proxies and anti-bot handling, which the README assigns to the separate hosted API rather than the open-source library
  • Users who cannot pay for or run an LLM, since the library depends on one for extraction
Step-by-step guide: how to use ScrapeGraphAIInstall, first run, examples, tips and FAQs
ScrapeGraphAI alternatives

How to get it

  1. 1

    Install with a package manager

    You'll need: Python · Playwright browser binaries (installed via playwright install) · An LLM: an API key for a provider such as OpenAI, or Ollama for local models

    Install the library

    pip install scrapegraphai

    Install browser dependencies for fetching websites

    playwright install
  2. 2

    Run it

    • Install the library and run playwright install
    • Create a graph_config with your chosen LLM (for example ollama/llama3.2 or openai/gpt-4o-mini)
    • Create a SmartScraperGraph with a prompt and a source URL
    • Call run() and print the returned result
  3. 3

    Get the most out of it

    • Use the multi-page graphs (SmartScraperMultiGraph, SearchGraph) to apply one prompt across many sources
    • Switch the llm section of the config to change providers without changing the rest of the code
    • Use the verbose and headless settings in the config to watch or hide the browser while debugging
    • Connect it to LangChain, LlamaIndex, or CrewAI using the integrations listed in the README

Downloads

Source code (.zip)main branch · straight from GitHubRelease v2.3.1Release notes and sourceOfficial website & docsscrapegraphai.com

Strengths

  • Extraction is driven by a natural-language prompt rather than hand-written selectors
  • Supports many LLM providers, including OpenAI, Groq, Azure, Gemini, and local models via Ollama
  • Released under the MIT license and has a large community with 32,000+ stars

Trade-offs

  • Results depend on the chosen LLM and can vary, and the README does not describe accuracy guarantees
  • Requires extra setup such as running playwright install to fetch website content and, for local models, installing Ollama
  • The README pushes a separate hosted product in several places, so readers should check which option fits their needs

How it compares

ProjectStarsTypeLevelUpdatedBest for
Scrapegraph-ai (this)32.4kLibraryintermediate1 months agoDevelopers who want to extract structured data from web pages using a plain-language prompt
spider-flow11.4kAppintermediate3.4 years agoNon-programmers who need to collect data from websites and prefer visual flow definitions
docling70kLibrarybeginner5 days agoPython developers building retrieval or RAG pipelines over PDFs and office files
llama_index52.8kLibraryintermediate10 days agoPython developers building retrieval-augmented generation (RAG) apps over private documents
LightRAG40.5kLibraryintermediate7 days agoDevelopers building question-answering over their own document collections
PageIndex40kLibraryintermediate7 days agoDevelopers building question-answering over long PDFs such as annual reports or manuals

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.