Skip to content

README.md

Latest commit

 

History

History
196 lines (132 loc) · 7.42 KB

README.md

File metadata and controls

196 lines (132 loc) · 7.42 KB
Raw

🚀 proxy-scraper-checker

CI

TUI Demo

A lightning-fast, feature-rich proxy scraper and checker built in Rust.

Collects HTTP/SOCKS4/SOCKS5 proxies from any number of sources, verifies each one really works, and writes the survivors out with response times, geolocation and network ownership attached.

✨ Key Features

  • ⚡ Fast — async Rust; a single binary for Windows, Linux, macOS and Android, with nothing to install
  • 🎯 Parses anything — proxies are pattern-matched out of raw text, HTML or JSON, so any source works untouched. Understands scheme://user:pass@host:port and CIDR ranges, from URLs or local files
  • ✅ Verifies, not just collects — every proxy has to fetch a real URL in full to survive, so ones that connect and then stall are dropped. Results are deduplicated across sources
  • 🌍 Rich metadata — response time, exit IP, ASN and city-level geolocation from offline MaxMind databases, with no per-proxy API calls
  • 📊 Interactive TUI — live per-protocol progress and logs
  • 📤 Flexible output — JSON with metadata, plus ready-to-paste plain text lists

🔗 Related

Want proxies without running anything? monosans/proxy-list is updated regularly using this tool.

⚠️ Safety warning

Checking opens hundreds of simultaneous connections to untrusted hosts. Your ISP may read that as abusive traffic, and cheap routers can drop connectivity when their NAT table fills. Consider a VPN, and lower max_concurrent_checks if a run destabilizes your network.

🚀 Quick start

Every option is documented inline in config.toml. The binary reads config.toml from the current directory; set PROXY_SCRAPER_CHECKER_CONFIG to point it somewhere else.

💻 Download & run a pre-built binary (easiest)

Note: Android/Termux users, see the dedicated section below.

  1. Download the archive for your platform from nightly builds

    • Look for artifacts starting with proxy-scraper-checker-binary- followed by your platform
    • Not sure which platform? Check the platform support table

  2. Extract the archive to a dedicated folder

  3. Configure by editing config.toml

  4. Run the executable

🐳 Docker

Note: The Docker image logs to stdout instead of showing the interactive TUI.

  1. Install Docker Compose

  2. Download the archive from nightly builds

    • Look for artifacts starting with proxy-scraper-checker-docker- followed by your CPU architecture

  3. Extract to a folder and edit config.toml

  4. Build and run:

    # Windows
docker compose build
docker compose up --no-log-prefix --remove-orphans

# Linux/macOS
docker compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g)
docker compose up --no-log-prefix --remove-orphans

Results land in ./out on the host; output.path is ignored in Docker.

📱 Android / Termux

Important: Install Termux from F-Droid, not Google Play (why?).

  1. Install with one command:

    bash <(curl -fsSL 'https://raw.githubusercontent.com/monosans/proxy-scraper-checker/main/termux.sh')

  2. Configure using a text editor:

    nano ~/proxy-scraper-checker/config.toml

  3. Run:

    cd ~/proxy-scraper-checker && ./proxy-scraper-checker
🛠️ Build from source

  1. Install the Rust toolchain

    See https://rust-lang.org/tools/install/

  2. Clone the repository

    git clone https://github.com/monosans/proxy-scraper-checker.git
cd proxy-scraper-checker

  3. Build a release binary with the TUI enabled

    cargo build --features tui --release --locked

  4. Run with the TUI

    cargo run --features tui --release --locked

Binary location:

  • target/release/proxy-scraper-checker (Linux/macOS)
  • target\release\proxy-scraper-checker.exe (Windows)

📤 Output

out/
├── proxies.json          working proxies with metadata, compact
├── proxies_pretty.json   the same data, indented
└── proxies/
    ├── all.txt           every proxy, with its protocol prefix
    ├── http.txt          one file per enabled protocol, no prefix
    ├── socks4.txt
    └── socks5.txt

{
  "protocol": "socks5",
  "username": null,
  "password": null,
  "host": "1.2.3.4",
  "port": 1080,
  "timeout": 0.42,
  "exit_ip": "5.6.7.8",
  "asn": {
    "autonomous_system_number": 12345,
    "autonomous_system_organization": "Example Networks"
  },
  "geolocation": {
    "country": { "iso_code": "US", "names": { "en": "United States" } },
    "city": { "names": { "en": "Chicago" } },
    "location": {
      "latitude": 41.85,
      "longitude": -87.65,
      "time_zone": "America/Chicago"
    }
  }
}

timeout is how long the whole request took, in seconds — connection, request and the complete response, which is also what sort_by_speed orders by. exit_ip is the address the target site actually saw — compare it with host to spot proxies that forward through somewhere else.

exit_ip, asn and geolocation are only filled in when check_url returns the exit IP, so pointing it at an ordinary web page leaves all three null. geolocation mirrors the GeoLite2 City record (abbreviated above); fields the database has no data for are omitted, and names are English only.

Stopping a run early — ESC or q in the TUI, Ctrl-C anywhere — writes out everything checked so far. If you stop before any proxy has been checked, the previous run's files are left untouched rather than emptied.

💎 Sponsors
🌐 RapidProxy.io

🚀 Want to be featured here? Support the project and get your name in this section.

💖 Support this project

  • ⭐ Star the repository — help others discover this tool
  • 🤝 Interested in sponsoring? DM me on Telegram

📄 License

MIT

This product includes GeoLite2 Data created by MaxMind, available from https://www.maxmind.com