Skip to content
Get packs

ProxyPool

Python tool that collects free proxy IPs from public sites, checks them, and serves working ones via an API.

GitHub Download 2.4.1
Stars
23.8k
Forks
5.4k
Licence
MIT
Language
Python
Last update
5 months ago
Type
App

What it is

ProxyPool is a Python service that regularly gathers free proxy addresses published on public websites, tests whether they still work, and stores the working ones in a Redis database. Other programs can then ask its HTTP API for a random proxy, which is useful for web scrapers that need to rotate IP addresses. It also offers a command-line mode and lets you add your own proxy sources.

  • Run a scheduled job that fetches and validates free proxies into Redis
  • Call the /get endpoint from a spider to obtain a random proxy and retry requests through it
  • Use /delete to remove a proxy that failed, keeping the pool cleaner
  • Add a custom proxy source by writing a small fetcher class in fetcher/sources/

Great for

  • Developers building web scrapers who want to experiment with rotating proxies
  • Hobbyists learning how proxy pools and scheduled crawling work
  • Teams who want a self-hosted proxy pool they can extend with their own sources

Not ideal for

  • Production scraping that needs reliable proxies, because the README notes free proxies have limited quality and recommends paid providers
  • Anyone needing anonymity or security guarantees, since the README does not discuss them and free proxies may be untrustworthy
  • Non-developers, since setup requires Python, Redis and configuration editing
Step-by-step guide: how to use ProxyPoolInstall, first run, examples, tips and FAQs
ProxyPool vs Spider-Flow

How to get it

  1. 1

    Clone and build from source

    You'll need: Python 3.8 to 3.11 (per README badges) · Redis

    Clone the repository

    git clone https://github.com/jhao104/proxy_pool.git

    Install dependencies

    pip install -r requirements.txt
  2. 2

    Run it

    • Edit setting.py to set DB_CONN to your Redis connection string
    • Start the scheduler with: python proxyPool.py schedule
    • In another terminal, start the API with: python proxyPool.py server
    • Open the API at http://127.0.0.1:5010 (the README also lists PORT = 5000 in its config sample, so check your setting.py) and request /get
  3. 3

    Get the most out of it

    • Use the ?type=https parameter on /get or /all to filter for HTTPS-capable proxies
    • Disable weak sources by adding their name to PROXY_FETCHER_EXCLUDE in setting.py
    • Run python proxyPool.py fetcher to list currently enabled proxy sources
    • Wrap the /get and /delete calls in a retry function, as the README's example shows
    • Use the included docker-compose file for a containerized setup

Downloads

Source code (.zip)master branch · straight from GitHubRelease 2.4.1Release notes and sourceOfficial website & docsjhao104.github.io/proxy_pool/

Strengths

  • Covers both scheduled collection and validation, so the pool refreshes automatically
  • Simple HTTP API (/get, /pop, /all, /count, /delete) that any language can call
  • Extensible: new proxy sources are added as Python files without changing core config

Trade-offs

  • Free proxy sources vary in quality and availability, as the README itself warns
  • Requires Redis and a Python environment, so setup has some moving parts
  • The README is partly in Chinese and includes promotional content for a paid proxy vendor

How it compares

ProjectStarsTypeLevelUpdatedBest for
proxy_pool (this)23.8kAppintermediate5 months agoDevelopers building web scrapers who want to experiment with rotating proxies
spider-flow11.4kAppintermediate3.4 years agoNon-programmers who need to collect data from websites and prefer visual flow definitions
examples-of-web-crawlers14.7kLearning resourcebeginner1.4 years agoBeginners who want to see working examples of web scraping in Python
hetty12.5kAppintermediate4 months agoSecurity researchers testing web applications
mitmproxy46.6kDev toolintermediate4 days agoDevelopers debugging how an app talks to a backend API
postgrest27.7kAppintermediate6 days agoDevelopers who already have a PostgreSQL database and want an HTTP API on top of it quickly

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.