Self-hosted OSINT automation platform. Forked from SpiderFoot with a modernized UI, overhauled module ecosystem, and improved developer experience.
- Frontend rewrite: CherryPy/jQuery/Bootstrap replaced with Flask/Tailwind CSS/HTMX/Alpine.js
- Dark SOC-themed UI with collapsible sidebar, category-grouped results, and HTMX-driven interactions
- Module audit: 11 dead modules removed, 6 broken APIs fixed, 22 new modules added
- 244 total modules covering passive recon, breach data, threat intel, cloud discovery, and vulnerability scanning
- BBOT integration: 4 wrapper modules providing access to 50+ BBOT sources
- Full REST API for programmatic scan management and data export
docker build -t spiderfoot-revival .
docker run -p 5001:5001 spiderfoot-revivalVisit http://localhost:5001
The default image is the slim Alpine build — Python deps only. To get the local-tool modules (BBOT, nmap, nuclei, dnstwist, whatweb, retire, testssl.sh, etc.) build the full image instead:
docker build -f Dockerfile.full -t spiderfoot-full .
docker run -p 5001:5001 -v /my/data:/var/lib/spiderfoot spiderfoot-fullFor BBOT active port scanning (masscan SYN scans) add --cap-add=NET_RAW; without it BBOT falls back to TCP-connect scans.
docker run -p 5001:5001 -v /my/data:/var/lib/spiderfoot spiderfoot-revivalpip install -r requirements.txt
python sf.py -l 127.0.0.1:5001Requires Python 3.9+.
The AI Assistant uses Server-Sent Events (SSE) to stream summaries from OpenRouter. Configure your key at Settings → AI Assistant and enable the feature.
- Single-worker assumption. The in-flight concurrency lock is process-local. Multi-worker deployments may permit duplicate concurrent generations for the same scan/model (each worker grants its own lock); we recommend single-worker until v2.
- gunicorn: SSE requires
--worker-class gevent(oreventlet), or sync workers with--threads N(≥4). The default sync single-thread worker will serialize all SSE clients and the UI will appear to hang. - nginx / Cloudflare: the streaming endpoint emits
X-Accel-Buffering: no. nginx honors this; Cloudflare requires the Enterprise plan. On Cloudflare Free, summaries still work but appear as a single chunk after the model finishes rather than streaming live. - Disable feature: uncheck Enable AI features in the AI Assistant settings to remove all AI UI elements.
By default, SpiderFoot runs without authentication. To enable it, create a passwd file:
# Create the passwd file with a bcrypt-hashed password
python -c "import bcrypt; print('admin:' + bcrypt.hashpw(b'changeme', bcrypt.gensalt()).decode())" > ~/.spiderfoot/passwdOn startup, SpiderFoot loads this file and enforces HTTP Basic Auth on all endpoints. Plaintext passwords in existing passwd files are automatically upgraded to bcrypt on first load.
- 244 OSINT modules across passive recon, active scanning, breach data, and threat intelligence
- Event-driven pipeline — modules produce and consume typed events, cascading discovery automatically
- Web UI — dark theme, real-time scan progress, categorized results, expandable event details
- REST API — full scan lifecycle management, JSON/CSV/GEXF export, config import/export
- CLI mode — run scans without the web server
- Correlation engine — YAML-configurable rules (44 built-in) for cross-referencing findings
- Docker-first — Alpine-based image, non-root user, persistent volume support
SpiderFoot can scan:
- Domain / subdomain names
- IP addresses
- Network subnets (CIDR)
- Email addresses
- Phone numbers
- Usernames
- Person / company names
- Bitcoin addresses
- ASNs
| Module | Source | Auth |
|---|---|---|
| Shodan InternetDB | Ports, CVEs, hostnames | Free, no key |
| LeakCheck | Breach database | Free public + pro |
| Hudson Rock | Infostealer intelligence | Free |
| Criminal IP | Threat intel | Free tier with key |
| Netlas.io | Attack surface search | Free tier with key |
| Validin | DNS history, CT logs | Free tier with key |
| OpenSanctions | PEP/sanctions screening | Free non-commercial |
| WhoisXML API | WHOIS, DNS, reverse IP | 500 free credits |
| BeVigil | Mobile app OSINT | Free credits |
| Postman | Workspace leak detection | Free, no key |
| ZoomEye | Internet scanning | Paid |
| FOFA | Internet asset search | Paid |
| Snusbase | Breach/credential search | Paid |
| BBOT Subdomain Enum | 50+ passive sources | Free |
| BBOT Active Scan | Ports, HTTP, SSL, fingerprinting | Free |
| BBOT Vuln Scanner | Nuclei, badsecrets, baddns | Free |
| BBOT Cloud Recon | S3/Azure/GCP buckets | Free |
| User Scanner | Email + username checking | Free |
| MISP | Threat intel sharing | Self-hosted |
| RansomLook | Ransomware victim tracking | Free, no key |
| C2-Tracker | Live C2 server feeds | Free, no key |
| Vulners | CVE/exploit database | Free tier with key |
| Module | Source | Auth |
|---|---|---|
| Ransomware.live | Ransomware leak-site victim lookup | Free, no key (rate-limited, personal use) |
ThreatCrowd, CRXcavator, F-Secure Riddler, Clearbit, RiskIQ, BitcoinAbuse, PunkSpider, Crobat API, Twitter, Venmo, MySpace — all dead or deprecated services.
DNSDumpster (new JSON API), DNSGrep (bufferover.run), Skymem (URL update), HackerTarget (rate limits), HaveIBeenPwned (v3 only), Keybase (deprecation notice).
SpiderFoot exposes a full REST API for integration with other tools:
# Start a scan
curl -X POST http://localhost:5001/api/startscan \
-d "scanname=test&scantarget=example.com&modulelist=sfp_dnsresolve,sfp_shodaninternetdb"
# List all scans
curl http://localhost:5001/api/scanlist
# Get scan results
curl http://localhost:5001/api/scaneventresults?id=SCAN_ID
# Export as JSON
curl http://localhost:5001/api/scanexportjsonmulti?ids=SCAN_ID
# Stop a scan
curl -X POST http://localhost:5001/api/stopscan?id=SCAN_IDSee CLAUDE_TECHNICAL.md for the full API reference.
sf.py # Entry point (CLI + web server)
sflib.py # Core library facade (delegates to net/*)
sfscan.py # Scan engine and module orchestration
sfcli.py # CLI client
modules/ # 244 OSINT modules (sfp_*.py)
tailwind.config.js # Tailwind PostCSS build config + safelist
spiderfoot/
app.py # Flask app factory, auth, CSRF
db.py # SQLite database layer
plugin.py # Base plugin class
correlation.py # YAML-based correlation engine
event.py # SpiderFootEvent class
target.py # SpiderFootTarget class
helpers.py # SpiderFootHelpers utility class
logger.py # SQLite log handler, queue listener
threadpool.py # SpiderFootThreadPool thread pool
net/ # Network utilities (extracted from sflib.py)
http.py # HTTP client (fetchUrl, sessions, proxy)
dns.py # DNS resolution and validation
ssl.py # Certificate parsing, safe sockets
host.py # IP/hostname/domain validation utilities
services/
event_service.py # Event formatting, categories, badge colors
preset_service.py # Scan preset catalog + DB seeding
ai_service.py # OpenRouter chat completions client
ai_persistence.py # AI summary DB persistence
blueprints/
api.py # REST API endpoints (/api/*)
ui.py # HTML page routes
fragments.py # HTMX fragment routes (/frag/*)
ai.py # AI summarization SSE routes (/frag/scan/*, /frag/correlation/*)
templates/ # Jinja2 templates (pages, components, fragments)
static/ # CSS, JS, vendor libs
correlations/ # 44 YAML correlation rules
| Layer | Technology |
|---|---|
| Backend | Python 3, Flask, SQLite |
| Frontend | Tailwind CSS (PostCSS build), HTMX, Alpine.js |
| Templates | Jinja2 with component/fragment pattern |
| Deployment | Docker (Alpine 3.18), non-root |
The local-tool modules (sfp_tool_*) shell out to external CLI binaries. Use Dockerfile.full to get all of them pre-installed, or install them yourself for local development:
- bbot —
pip install bbot && bbot --install-all-deps -y(the second step pulls down per-module deps; without it bbot will try to invokesudoat scan time) - nmap, nbtscan, onesixtyone — system package manager
- nuclei, fingerprintx — Go binaries from ProjectDiscovery / fullhunt-io releases
- whatweb — Ruby;
gem install whatwebor clone from upstream - retire —
npm install -g retire - testssl.sh, CMSeeK — git clone the upstream repos
- dnstwist, wafw00f, snallygaster, trufflehog —
pip install <name>
API keys and settings are configured through the web UI at /opts (Settings page). Settings can also be imported/exported:
# Export settings
curl http://localhost:5001/api/optsexport > config.cfg
# Import settings
curl -X POST http://localhost:5001/api/savesettings -F "configFile=@config.cfg"- Fork the repository
- Create a feature branch
- Follow the module pattern in CLAUDE.md — every module needs
meta,opts,optdescs,setup(),watchedEvents(),producedEvents(),handleEvent() - Use
self.sf.fetchUrl()for all HTTP requests (never importrequestsdirectly) - Test via Docker:
docker build -t sf-test . && docker run -d -p 5001:5001 sf-test - Submit a pull request
- CHANGELOG.md — Release history
- CLAUDE.md — Project conventions and development guide
- CLAUDE_TECHNICAL.md — Deep technical reference (architecture, API, DB schema)
- correlations/README.md — Correlation rule authoring guide
MIT — see LICENSE.
Originally created by Steve Micallef. This fork is maintained by @boredchilada.