Spider-Flow
A Java-based crawler platform where you build scraping flows as diagrams instead of code.
- Stars
- 11.4k
- Forks
- 2.2k
- Licence
- MIT
- Language
- Java
- Last update
- 3.4 years ago
- Type
- App
No commits in over a year — check for forks or alternatives before adopting.
What it is
spider-flow is a web scraping platform that lets you define a crawler as a flowchart in a graphical editor rather than writing code. The README describes it as highly flexible and configurable, with support for extracting data via XPath, JsonPath, CSS selectors, and regex, saving results to databases or files, and handling JavaScript-rendered pages. It is written in Java and released under the MIT license.
- Collect product or listing data from pages that load content with JavaScript
- Extract fields from JSON or XML API responses and store them in a SQL database
- Run a crawler on a schedule and review its task logs and debug output
- Extend the platform with a plugin such as the Selenium or Redis plugin listed in the README
Great for
- Non-programmers who need to collect data from websites and prefer visual flow definitions
- Java developers who want an existing crawler platform to extend with plugins
- Teams that need scraped data written directly to a database or file
- Users who need scheduled task monitoring and per-task logs for crawlers
Not ideal for
- Projects that need a lightweight script rather than a full platform, since it runs as a server-side application with a UI, database, and plugins
- Anyone scraping sites in ways that break the law or the site's terms; the README's disclaimer explicitly prohibits illegal use
- Users who need a fully maintained project: the last push shown is from June 2023 and the latest release is v0.5.0 with no binaries attached
How to get it
- 1
Clone and build from source
You'll need: JDK 1.8 or newer
Clone the repository
git clone https://github.com/ssssssss-team/spider-flow.git - 2
Run it
- Clone the repository and build it with a JDK 1.8+ environment, following the full setup guide at spiderflow.org
- Start the application using the instructions in the documentation
- Open the web interface and create a new crawler flow in the graphical editor
- Use the built-in test and debug views to check the flow before running it
- 3
Get the most out of it
- Add the Selenium plugin to scrape pages that need a real browser
- Use the proxy pool plugin when a target site restricts access by IP
- Use the MongoDB or OSS plugins to save results to other storage
- Use the custom function and custom executor features to add your own logic
- Check the task logs to troubleshoot runs
Downloads
Strengths
- Visual flow editor lets users build crawlers without writing code, according to the README
- Supports many extraction methods (XPath, JsonPath, CSS selectors, regex) and several data formats
- Includes task monitoring, logs, cookie management, and proxy support
Trade-offs
- The latest release has no binaries, so you will likely need to build from source; the README does not give full build steps in the excerpt provided
- Last code push is dated June 2023, so maintenance status is unclear
- The README is partly in Chinese and the excerpt does not include complete English setup documentation
How it compares
| Project | Stars | Type | Level | Updated | Best for |
|---|---|---|---|---|---|
| spider-flow (this) | 11.4k | App | intermediate | 3.4 years ago | Non-programmers who need to collect data from websites and prefer visual flow definitions |
| proxy_pool | 23.8k | App | intermediate | 5 months ago | Developers building web scrapers who want to experiment with rotating proxies |
| katana | 17.8k | CLI | intermediate | 4 days ago | Security testers mapping the pages and endpoints of a site they are authorized to test |
| examples-of-web-crawlers | 14.7k | Learning resource | beginner | 1.4 years ago | Beginners who want to see working examples of web scraping in Python |
| Scrapegraph-ai | 32.4k | Library | intermediate | 1 months ago | Developers who want to extract structured data from web pages using a plain-language prompt |
| jsoup | 11.4k | Library | intermediate | 8 days ago | Java developers who need to scrape or extract data from web pages |
Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.