Verify hosts are alive before emitting wayback URLs

[liquidsec]

Problem

With urls=True, wayback emits a URL_UNVERIFIED for every URL that survives filtering, with no check that the host still exists. A CDX response for a large domain routinely contains hundreds of hosts that are long dead, and every one of their URLs gets emitted for the rest of the scan to chase: http requests each one, excavate parses the results, the spider follows them. That cost is large and entirely wasted.

Nothing in the emission path checked liveness. _pre_process_urls applies an in-scope test, but only for the archive/parameter/interesting-file metadata; the URL_UNVERIFIED list bypassed it. abort_if runs after DNS resolution but only requires the in-scope tag, which an unresolvable host still carries.

Change

Probe each (scheme, netloc) once with HEAD / before emitting any of its URLs, and drop the URLs of hosts that don't answer.

• Batched through request_batch_stream at 25 concurrency, follow_redirects=False, so it's one request per host regardless of URL count.
• Any response counts as alive, including 4xx/5xx. Only connect failures and timeouts mark a host dead.
• Explicit retries=1 rather than inheriting web.http_retries, so a single dropped packet can't condemn a live host even if a user sets that to 0.
• Verdicts cached in an LRUCache(maxsize=40000), measured at 9.44 MB when full with realistic hostnames (248 bytes/entry). The cache lives on the module instance, so that figure is per scan.
• Skipped entirely when a proxy is configured. The proxy answers on the target's behalf, so an unreachable upstream comes back as a proxy 502 that's indistinguishable from a real one. Rather than let proxy config flip the verdict for the same target, no probe is sent and every host's URLs are emitted.

Only reachable from the urls=True branch, so the default config sends no probes.

The archive path is deliberately unaffected

_archive_cache is populated before the liveness gate. Dead hosts are exactly what the archive feature exists for, so finish() still fetches every snapshot it would have before. Eviction behavior is unchanged too: it only fires on a live 2xx URL event, which a dead host never produced anyway. The six existing archive tests all run against http://127.0.0.1:1/..., i.e. dead hosts, and pass unchanged.

_parameter_cache is now only populated for live hosts. Those entries could only ever be read when a live 2xx URL event arrived, so nothing is lost.

Interesting-file findings are deliberately unaffected

Same reasoning as the archive path, and worth stating explicitly since the gate sits between them. interesting_files is built in _pre_process_urls, and _check_interesting_files fetches from web.archive.org, not from the host. A backup.zip that no longer exists on a dead host is exactly what makes the archive worth searching, so those FINDINGs are emitted regardless of liveness.

Pinned by TestWaybackDeadHostInterestingFile, which asserts a dead host emits no URL_UNVERIFIED events but still produces the finding.

Tests

New TestWaybackDeadHostSkip: two hosts with identical DNS, only one answering HTTP. Asserts only the live host's URLs are emitted, that the host is probed exactly once regardless of URL count, and the resulting cache contents.

Seven existing tests needed a liveness mock, since their hosts are mock-intercepted and an unmatched URL now reads as dead. TestWaybackParameters got an explicit / handler rather than relying on the httpserver's 500-fallback.

22 passed, 0 failed.

Notes

• Verdicts are cached for the whole scan, so a host that's briefly unreachable at probe time loses its URLs for the rest of the run. The retry is the mitigation.
• http-only wayback records for a host that has since moved to HTTPS-only now get dropped rather than emitted. They were already dead ends, since http would have failed the same connection.
• This makes the module send target traffic when urls=True while it's still flagged passive. That's a pre-existing inconsistency (_is_http_wildcard_host already probes targets in every config) and is tracked separately in Module flags are static, but some modules are active or passive depending on config #3358 rather than fixed here.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs wayback-host-liveness (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	3.75ms	4.26ms	+13.6% 🟡🟡	⚠️
Bloom Filter Large Scale Dns Brute Force	15.17ms	17.75ms	+17.0% 🟡🟡	⚠️
Large Closest Match Lookup	361.20ms	367.38ms	+1.7% ⚪	✅
Realistic Closest Match Workload	196.64ms	195.58ms	-0.5% ⚪	✅
Event Memory Medium Scan	1403 B/event	1403 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	408.15ms	409.43ms	+0.3% ⚪	✅
Event Validation Full Scan Startup Large Batch	551.89ms	546.56ms	-1.0% ⚪	✅
Make Event Autodetection Small	26.43ms	26.18ms	-0.9% ⚪	✅
Make Event Autodetection Large	271.28ms	269.34ms	-0.7% ⚪	✅
Make Event Explicit Types	11.80ms	11.69ms	-0.9% ⚪	✅
Excavate Single Thread Small	4.337s	4.267s	-1.6% ⚪	✅
Excavate Single Thread Large	10.062s	10.044s	-0.2% ⚪	✅
Excavate Parallel Tasks Small	4.461s	4.482s	+0.5% ⚪	✅
Excavate Parallel Tasks Large	6.497s	6.368s	-2.0% ⚪	✅
Intercept Throughput Small	887.67ms	878.73ms	-1.0% ⚪	✅
Intercept Throughput Medium	876.52ms	880.31ms	+0.4% ⚪	✅
Dns Throughput Quiet	2.820s	2.815s	-0.2% ⚪	✅
Dns Throughput Loaded	1.853s	1.824s	-1.6% ⚪	✅
Dns Throughput Inherited	2.551s	2.618s	+2.6% ⚪	✅
Is Ip Performance	2.33ms	2.26ms	-2.8% ⚪	✅
Make Ip Type Performance	239.52µs	239.54µs	+0.0% ⚪	✅
Mixed Ip Operations	2.40ms	2.36ms	-1.7% ⚪	✅
Memory Use Web Crawl	1.0 MB	1.0 MB	+1.0% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	-0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	11.5 MB	10.3 MB	-10.5% 🟢🟢	🚀
Scan Throughput 100	2.949s	2.932s	-0.6% ⚪	✅
Scan Throughput 1000	20.483s	20.314s	-0.8% ⚪	✅
Typical Queue Shuffle	5.43µs	5.42µs	-0.1% ⚪	✅
Priority Queue Shuffle	25.91µs	26.02µs	+0.4% ⚪	✅

🎯 Performance Summary

+ 1 improvement 🚀
! 2 regressions ⚠️
  28 unchanged ✅

🔍 Significant Changes (>10%)

• Bloom Filter Dns Mutation Tracking Performance: 13.6% 🐌 slower
• Bloom Filter Large Scale Dns Brute Force: 17.0% 🐌 slower
• Memory Use Parallel Chains: 10.5% 🚀 less memory

---

🐍 Python Version 3.11.16

[codecov]

Codecov Report

❌ Patch coverage is 94.68085% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (3be13de) to head (1ffd801).
⚠️ Report is 36 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/modules/wayback.py	93%	3 Missing ⚠️
...st/test_step_2/module_tests/test_module_wayback.py	97%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #3359    +/-   ##
======================================
+ Coverage     90%     90%    +1%     
======================================
  Files        450     454     +4     
  Lines      46327   47177   +850     
======================================
+ Hits       41588   42406   +818     
- Misses      4739    4771    +32     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[singlerider]

Good change. Gating on (scheme, netloc) is the right granularity. Full wayback suite passes on the branch: 20 in 191s.

PR body is solid: the _archive_cache ordering, the _parameter_cache reachability argument, and punting the passive/active flag to #3358 all saved me work.

Two blockers inline.

[singlerider]

🔴 Blocking. response is not None conflates "host answered" with "something in the path answered." With a proxy configured (web.py:128 applies it), a 502 for an unreachable upstream reads live while a direct connect failure reads dead. Same target, opposite verdict from proxy config alone.

Also: the 40K LRUCache is per module instance per scan, so 9.44 MB is per-scan.

[liquidsec]

Both fixed.

Proxy: liveness probing is now skipped entirely when web.http_proxy is set. The proxy answers on the target's behalf, so there's no verdict worth trusting and we emit everything rather than guess. Worth noting this doesn't change what gets emitted under a proxy, since a proxy 502 already read as live; it just drops a useless probe per host and stops making a claim the data doesn't support. Covered by TestWaybackProxySkipsLivenessProbe.

Interesting files: no behavior change, dead hosts emitting those findings is the point of the feature. A backup.zip that's gone from the host but still on archive.org is exactly what it's for. Pinned with TestWaybackDeadHostInterestingFile (dead host, no URL_UNVERIFIED, finding still emitted) and documented in the PR body next to the archive exemption.

Also corrected the cache figure in the body to say per scan. 22 passed.

[singlerider]

Re-reviewed at 1ffd801, both blockers resolved. Full wayback suite passes on the branch: 22 in 128s.

🟢 Proxy fix

Skipping the probe entirely is the right call over trying to interpret a proxied response. TestWaybackProxySkipsLivenessProbe asserts no probe is sent, the cache stays empty, and the URL is emitted anyway, which is the behavior that matters. RESOLVED ✅

🟢 Interesting-file exemption

Your reasoning is right and I withdraw the objection. TestWaybackDeadHostInterestingFile pins it exactly where it needed pinning: dead host, no URL_UNVERIFIED, finding still emitted. The PR body now states it next to the archive exemption. RESOLVED ✅

🔴 Blocking, _liveness_probing reads the raw config instead of the resolved proxy

setup() gates on self.scan.config.get("web", {}).get("http_proxy"), but the value that decides whether a request is actually proxied is WebHelper._http_proxy combined with _http_proxy_exclude (web.py:127-128). A user with http_proxy set and the target in http_proxy_exclude gets no probing at all, even though those probes would go direct and return a trustworthy verdict. Read the helper's resolved state rather than the raw config key.

🔴 Blocking, _live_netlocs runs per query call, not per scan

The LRUCache dedupes across calls, but the verbose line and the request_batch_stream dispatch are inside query(), so a scan with many in-scope domains issues a separate batch per domain at 25 concurrency each. On a large multi-domain scan that is a lot of concurrent probe batches against overlapping infrastructure. Worth either hoisting the batch or bounding it with the module's existing concurrency controls.

🟢 Test coverage

TestWaybackDeadHostSkip asserting the exact probe list, not just the emitted URLs, is what makes the once-per-host claim verifiable. Good test.