Set url_extension on HTTP_RESPONSE events

[liquidsec]

Problem

HTTP_RESPONSE.sanitize_data() overrides URL_UNVERIFIED.sanitize_data() without calling super() and without extracting the URL file extension. Since url_extension is declared in __slots__ with no default, it is never set on HTTP_RESPONSE events.

Paramminer's filter_event reads getattr(event, "url_extension", None), gets None, and lets every static-file URL through:

https://example.com/files/doc.pdf  ->  HTTP_RESPONSE.url_extension = None
                                       URL_UNVERIFIED.url_extension = 'pdf'

Every PDF, DOCX, ZIP etc. then receives the full paramminer treatment: 2 baseline requests plus a 0.5s sleep, then a binary search across the 5,289-word wordlist (~13 rounds of ~40-param batches). On wayback-heavy scans this dominates runtime and produces blasthttp timeouts. All three paramminer variants are affected, since they all watch HTTP_RESPONSE and share filter_event.

This is the same class of bug as 52c0cc8, which hoisted url_extension to DictEvent so WEB_PARAMETER events would get it. HTTP_RESPONSE was missed because it overrides sanitize_data and never picks up the inherited logic.

Fix

The extension logic was already duplicated between DictEvent and URL_UNVERIFIED, so it is extracted into DictEvent._set_url_extension() and used at all three call sites rather than adding a third copy.

The second half

Populating url_extension alone introduces a regression. url_extension_special: [js] is enforced in bbot/modules/base.py, and since 52c0cc8 that check applies to all event types rather than URL types only. Giving HTTP_RESPONSE the attribute newly drags it under that gate:

badsecrets on .js HTTP_RESPONSE
   before: (True,  'precheck succeeded')
   after:  (False, 'it has a special URL extension (js)...')

http fetches JS URLs (accept_url_special = True) and emits HTTP_RESPONSEs for them, which badsecrets and other content-analysis modules consume today. Without a guard they would silently stop.

HTTP_RESPONSE was only ever exempt from that gate by accident, because it lacked the attribute. _url_special_filterable makes the exemption explicit: events representing already-retrieved content stay out of a gate whose purpose (per defaults.yml: "URLs with these extensions are not distributed to modules") is deciding which URLs to hand out, not which response bodies to analyze.

Notes

retirejs and lightfuzz are unaffected. retirejs watches URL_UNVERIFIED, and lightfuzz's static filter is scoped to WEB_PARAMETER. Both types already had url_extension.

The existing TestParamminer_Getparams_filter_static passed before this fix, because it only asserted that no parameter was confirmed on the .pdf URL rather than that the work was skipped. It is strengthened here to assert the invariant, and both new tests fail without the one-line fix.

[codecov]

Codecov Report

❌ Patch coverage is 93.54839% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (a394d12) to head (9467e68).

Files with missing lines	Patch %	Lines
bbot/core/event/base.py	88%	2 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #3406   +/-   ##
=====================================
- Coverage     90%     90%   -0%     
=====================================
  Files        454     454           
  Lines      47081   47096   +15     
=====================================
+ Hits       42320   42329    +9     
- Misses      4761    4767    +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs http-response-url-extension (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	3.74ms	3.72ms	-0.5% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	15.34ms	15.37ms	+0.2% ⚪	✅
Large Closest Match Lookup	363.14ms	376.68ms	+3.7% ⚪	✅
Realistic Closest Match Workload	202.62ms	201.01ms	-0.8% ⚪	✅
Event Memory Medium Scan	1403 B/event	1402 B/event	-0.1% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	419.30ms	417.54ms	-0.4% ⚪	✅
Event Validation Full Scan Startup Large Batch	557.66ms	557.74ms	+0.0% ⚪	✅
Make Event Autodetection Small	27.26ms	27.79ms	+1.9% ⚪	✅
Make Event Autodetection Large	275.43ms	279.93ms	+1.6% ⚪	✅
Make Event Explicit Types	11.99ms	12.25ms	+2.2% ⚪	✅
Excavate Single Thread Small	4.649s	4.261s	-8.3% ⚪	✅
Excavate Single Thread Large	10.409s	10.258s	-1.4% ⚪	✅
Excavate Parallel Tasks Small	4.800s	4.456s	-7.2% ⚪	✅
Excavate Parallel Tasks Large	6.833s	6.591s	-3.5% ⚪	✅
Intercept Throughput Small	898.53ms	903.03ms	+0.5% ⚪	✅
Intercept Throughput Medium	958.20ms	888.17ms	-7.3% ⚪	✅
Dns Throughput Quiet	2.780s	2.814s	+1.2% ⚪	✅
Dns Throughput Loaded	1.987s	1.888s	-5.0% ⚪	✅
Dns Throughput Inherited	2.281s	2.445s	+7.2% ⚪	✅
Is Ip Performance	2.37ms	2.36ms	-0.3% ⚪	✅
Make Ip Type Performance	241.47µs	243.98µs	+1.0% ⚪	✅
Mixed Ip Operations	2.45ms	2.51ms	+2.4% ⚪	✅
Memory Use Web Crawl	1.0 MB	1.0 MB	+0.0% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	11.5 MB	12.1 MB	+5.2% ⚪	✅
Scan Throughput 100	2.870s	3.039s	+5.9% ⚪	✅
Scan Throughput 1000	22.053s	21.681s	-1.7% ⚪	✅
Typical Queue Shuffle	5.44µs	5.47µs	+0.6% ⚪	✅
Priority Queue Shuffle	26.26µs	26.16µs	-0.4% ⚪	✅

🎯 Performance Summary

✅ No significant performance changes detected (all changes <10%)

---

🐍 Python Version 3.11.16