Auto-Get-PY

Universal web scraper with media download capability, pluggable decryptors, and a local Web UI management dashboard.

AI-Generated Project — This codebase was generated by Claude Code (Anthropic) through iterative design, implementation, and review cycles. A human provided requirements and direction; the AI produced all code, tests, documentation, and commits.

---

Features

• Universal media downloading — Images (jpg, png, gif, webp, avif, heic...), Videos (mp4, mkv, webm, ts, m3u8...), Audio (mp3, flac, aac, ogg...), Documents (pdf, docx, epub...), Archives (zip, rar, 7z...)
• Pluggable decryptor pipeline — Base64, Hex, AES (CBC/ECB/GCM), XOR, ROT47, URL sign stripping, custom Python expressions
• Async concurrent downloads — Configurable concurrency, chunked transfer with progress tracking
• Resume support — HTTP Range header for interrupted downloads
• Automatic retry — Exponential backoff (1s, 2s, 4s...), configurable max attempts
• Dark-themed Web UI — SPA dashboard with real-time WebSocket progress, task management, file browser, settings
• SQLite persistence — Tasks, downloads, and settings survive restarts
• Single-command launch — python app.py starts everything

---

Quick Start

Prerequisites

• Python 3.10+
• pip

Install

git clone https://github.com/Hotsteel2901/Auto-Get-PY.git
cd Auto-Get-PY
pip install -r requirements.txt

Run

python app.py
# or with custom port:
python app.py --port 9090
# or with auto-reload (dev mode):
python app.py --reload

Open http://localhost:8000 in your browser.

---

Web UI Guide

The sidebar has 5 pages:

1. Dashboard

The landing page. Shows:

• Stat cards — Running / Completed / Failed task counts
• Recent tasks — Table with progress bars, pause/resume/retry buttons
• Live updates — All stats refresh automatically via WebSocket as downloads progress

2. New Task

Create a scraping task:

Section	Description
Task Name	A label for this task
Target URL	The page to scrape for media links
File Type Filters	Check/uncheck extensions by category (Images / Videos / Audio / Documents / Archives). Custom extensions at the bottom.
Decryptors	Enable decryption layers. AES and XOR have expandable config panels for keys. Custom lets you type a Python expression.
Advanced Options	Concurrency (1-20), request delay, timeout, max retries, max file size, output directory
Custom Headers	Add HTTP headers (e.g., Referer, User-Agent, Authorization)

Click Start Scraping — the task is created and immediately begins.

3. Downloads

Browse download results across all tasks:

• Filter by task via dropdown
• Search by filename
• Each row shows: filename, file size, status badge, progress bar, timestamp
• Download button for completed files
• Failed downloads show error messages

4. Settings

Global defaults persisted in SQLite:

• Default concurrency
• Default output directory
• AES key and IV (hex-encoded) for decryptor reuse across tasks

5. Files

Browse all downloaded files on disk. Sortable by modification time (newest first). Click Download to serve the file locally.

---

Decryptor Reference

Decryptors run in pipeline order (by priority). The first decryptor whose can_handle() check passes processes the content. The pipeline can iterate up to 3 passes.

Decryptor	Priority	Config	Use Case
Base64	10	None	Base64-encoded media URLs in page source
Hex	10	None	Hex-encoded strings
AES	20	key (hex), iv (hex), mode (CBC/ECB/GCM)	AES-encrypted content with known key
XOR	30	key (hex)	Simple XOR-obfuscated data
URL Sign	40	None (optional: extra param names)	Strips sign, token, expires etc. from URLs
ROT47	50	None	ROT47-encoded text in page source
Custom	100	Python expression	User-defined transformation. content is the bytes variable. Example: bytes(b ^ 0xFF for b in content)

Security note on Custom decryptor: The eval()-based custom expression runs unsandboxed in the Python process. It is intended for trusted local use only.

---

CLI Reference

python app.py [options]

Options:
  --host HOST     Bind host (default: 0.0.0.0)
  --port PORT     Bind port (default: 8000)
  --reload        Enable uvicorn auto-reload for development

Example workflows

# Basic usage
python app.py

# Accessible from LAN
python app.py --host 0.0.0.0 --port 8080

# Development with hot reload
python app.py --reload

---

API Overview

All endpoints are under /api/.

Tasks

Method	Path	Description
GET	/api/tasks	List tasks (query: status, offset, limit)
POST	/api/tasks	Create task ({name, url, config})
GET	/api/tasks/{id}	Get task detail
PUT	/api/tasks/{id}	Update task config
DELETE	/api/tasks/{id}	Delete task + cascade downloads
POST	/api/tasks/{id}/start	Start scraping
POST	/api/tasks/{id}/pause	Pause
POST	/api/tasks/{id}/resume	Resume
POST	/api/tasks/{id}/retry	Retry all failed downloads

Downloads

Method	Path	Description
GET	/api/tasks/{id}/downloads	List downloads for a task (query: status)

Settings

Method	Path	Description
GET	/api/settings	Get all settings
PUT	/api/settings	Update settings (key-value dict)

Files

Method	Path	Description
GET	/api/files	Browse downloaded files (query: dir)
GET	/api/files/download/{filename}	Serve file for local download

WebSocket

Path	Description
/ws/progress	Real-time progress: {task_id, done, total, current_file, speed}

---

Task Config Reference

The config field in task creation accepts this JSON structure:

{
  "concurrency": 5,
  "output_dir": "./downloads",
  "decryptors": ["base64", "aes"],
  "decryptor_opts": {
    "aes": {"key": "your-hex-key", "iv": "your-hex-iv", "mode": "cbc"}
  },
  "url_filters": {
    "include": ["*.jpg", "*.mp4", "*.pdf"],
    "exclude": ["*.gif"]
  },
  "custom_headers": {
    "Referer": "https://example.com",
    "User-Agent": "Mozilla/5.0 ..."
  },
  "request_delay_sec": 0.5,
  "request_timeout_sec": 30,
  "max_retries": 3,
  "max_file_size_mb": 500
}

---

Project Structure

Auto-Get-PY/
├── app.py                         # FastAPI entry + CLI
├── scraper/
│   ├── engine.py                  # Orchestrator: fetch → decrypt → extract → download
│   ├── extractor.py               # Media URL extraction from HTML
│   ├── downloader.py              # Async chunked download + resume
│   ├── task_manager.py            # Lifecycle, concurrency, pause/resume
│   └── decryptors/
│       ├── base.py                # BaseDecryptor ABC + registry + pipeline
│       ├── base64_dec.py          # Base64 decoder
│       ├── hex_dec.py             # Hex decoder
│       ├── aes_dec.py             # AES-CBC/ECB/GCM decoder
│       ├── xor_dec.py             # XOR decoder
│       ├── url_sign_dec.py        # URL signing param stripper
│       ├── rot47_dec.py           # ROT47 character shift
│       └── custom_dec.py          # User-defined Python expression
├── db/
│   ├── schema.py                  # SQLite schema + init_db
│   └── queries.py                 # Async query functions
├── api/
│   ├── tasks.py                   # Task CRUD + control routes
│   ├── downloads.py               # Download list route
│   ├── settings.py                # Settings get/put routes
│   ├── files.py                   # File browsing + serving
│   └── websocket.py              # WebSocket progress handler
├── webui/
│   ├── index.html                 # SPA shell
│   ├── css/style.css              # Dark theme design system
│   └── js/
│       ├── app.js                 # Router + WebSocket + utilities
│       ├── api.js                 # REST API client
│       ├── dashboard.js           # Dashboard page
│       ├── task-form.js           # New task form
│       ├── downloads.js           # Downloads browser
│       ├── settings.js            # Settings page
│       └── files.js               # Downloaded files browser
├── tests/                         # 29 passing tests
├── downloads/                     # Default output directory
└── requirements.txt               # Python dependencies

---

Dependencies

Package	Purpose
fastapi	Web framework + API
uvicorn	ASGI server
aiohttp	Async HTTP client for scraping/downloading
aiofiles	Async file I/O
aiosqlite	Async SQLite
pydantic	Request validation
pycryptodome	AES decryption
httpx	Test client

---

Testing

pip install httpx
python -m pytest tests/ -v

29 tests covering: all 7 decryptors, URL extractor, async downloader (mocked), API endpoints, task lifecycle.

---

Limitations & Scope

This is a local-first, single-user tool. The following are intentionally out of scope:

• Multi-user authentication / login
• Distributed crawling (Celery, Redis)
• Browser automation (Playwright/Selenium for JS-rendered pages)
• OCR-based captcha solving
• Custom expression sandboxing (trusted local use assumed)

---

License

MIT

---

中文版 (Chinese Version)

Generated by Claude Code (deepseek) (Anthropic) — May 2026