Add js_unpacker module for detecting and unpacking obfuscated JS

[liquidsec]

Summary

Adds a new js_unpacker module that detects and unpacks common JavaScript
packing/obfuscation formats using YARA rules, then re-emits the unpacked
body as an HTTP_RESPONSE so excavate (and everything downstream: badsecrets,
secret scanners, etc.) can extract URLs, credentials, JWTs, and API endpoints
that would otherwise be invisible inside the packed source.

What it handles

• source-map: fetches the referenced .map file, decodes sourcesContent,
  emits a FINDING when a source map with embedded sources is exposed
• Dean Edwards : eval(function(p,a,c,k,e,d){…}) packer
• obfuscator.io : string-array + rotation obfuscation
• Next.js : extracts routes from __BUILD_MANIFEST and __NEXT_DATA__
• webpack : webpackJsonp bundles

Emits URL_UNVERIFIED for Next.js routes directly; everything else flows
through excavate on the unpacked body.

Other changes

• HTTP_RESPONSE._data_id now includes the source module name. Without this,
  js_unpacker's unpacked re-emits (source-map, Dean Edwards, obfuscator.io,
  webpack - 4 of the 5 unpackers) dedupe against the original http response
  and never reach downstream modules.
• Add js_unpacker to the js-audit preset.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs js-unpacker (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	2.13ms	2.37ms	+11.2% 🟡🟡	⚠️
Bloom Filter Large Scale Dns Brute Force	9.05ms	10.47ms	+15.7% 🟡🟡	⚠️
Large Closest Match Lookup	221.27ms	237.19ms	+7.2% ⚪	✅
Realistic Closest Match Workload	123.59ms	125.90ms	+1.9% ⚪	✅
Event Memory Medium Scan	1402 B/event	1402 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	266.21ms	273.56ms	+2.8% ⚪	✅
Event Validation Full Scan Startup Large Batch	337.10ms	344.75ms	+2.3% ⚪	✅
Make Event Autodetection Small	14.99ms	15.13ms	+0.9% ⚪	✅
Make Event Autodetection Large	149.83ms	151.61ms	+1.2% ⚪	✅
Make Event Explicit Types	6.25ms	6.55ms	+4.9% ⚪	✅
Excavate Single Thread Small	3.003s	2.946s	-1.9% ⚪	✅
Excavate Single Thread Large	6.494s	6.414s	-1.2% ⚪	✅
Excavate Parallel Tasks Small	3.173s	3.120s	-1.7% ⚪	✅
Excavate Parallel Tasks Large	4.827s	4.775s	-1.1% ⚪	✅
Intercept Throughput Small	699.03ms	688.39ms	-1.5% ⚪	✅
Intercept Throughput Medium	695.21ms	751.58ms	+8.1% ⚪	✅
Dns Throughput Quiet	1.886s	1.856s	-1.6% ⚪	✅
Dns Throughput Loaded	1.268s	1.368s	+7.8% ⚪	✅
Dns Throughput Inherited	1.740s	1.602s	-7.9% ⚪	✅
Is Ip Performance	1.31ms	1.47ms	+11.9% 🟡🟡	⚠️
Make Ip Type Performance	117.77µs	130.48µs	+10.8% 🟡🟡	⚠️
Mixed Ip Operations	1.38ms	1.53ms	+10.6% 🟡🟡	⚠️
Memory Use Web Crawl	1.0 MB	1.0 MB	+1.0% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	11.1 MB	10.2 MB	-7.7% ⚪	✅
Scan Throughput 100	2.111s	2.183s	+3.4% ⚪	✅
Scan Throughput 1000	15.931s	15.715s	-1.4% ⚪	✅
Typical Queue Shuffle	3.59µs	3.64µs	+1.4% ⚪	✅
Priority Queue Shuffle	16.76µs	18.42µs	+9.9% ⚪	✅

🎯 Performance Summary

! 5 regressions ⚠️
  26 unchanged ✅

🔍 Significant Changes (>10%)

• Bloom Filter Dns Mutation Tracking Performance: 11.2% 🐌 slower
• Bloom Filter Large Scale Dns Brute Force: 15.7% 🐌 slower
• Is Ip Performance: 11.9% 🐌 slower
• Make Ip Type Performance: 10.8% 🐌 slower
• Mixed Ip Operations: 10.6% 🐌 slower

---

🐍 Python Version 3.11.16

[codecov]

Codecov Report

❌ Patch coverage is 90.39735% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (ba49607) to head (5675a79).

Files with missing lines	Patch %	Lines
bbot/modules/js_unpacker.py	88%	29 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #3335    +/-   ##
======================================
- Coverage     90%     90%    -0%     
======================================
  Files        454     456     +2     
  Lines      47081   47382   +301     
======================================
+ Hits       42324   42593   +269     
- Misses      4757    4789    +32     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[singlerider]

Tests pass, coverage is real: all 5 unpackers detected, end-to-end excavate extraction asserted, and the badsecrets chain test validates the _reemit_source dedup fix.

_data_id change is appropriately surgical (only diverges when _reemit_source is set), so normal http responses are unaffected.

Two nits:

• _reemit_source is stored in event.data and sanitize_data does not strip it, so it leaks into --json output. Suggest filtering underscore-prefixed keys in serialization.
• yara runs in run_in_executor_cpu (correct), but the per-unpacker DOTALL regexes run on the event loop on large minified JS. Consider offloading the heavy regex too.

LGTM.

[liquidsec]

@singlerider

Left _reemit_source as-is. It's an internal dedup marker, cosmetic in output only.

Fixed the regex point: each unpacker's CPU work now runs off the event loop via run_in_executor_cpu() / helpers.re. The load-bearing part is import re to import regex as re, since stdlib re holds the GIL for the whole match and a thread pool alone would offload nothing.

Also fixed a pre-existing flake in the test, ~10% of runs on Dean Edwards: excavate didn't find /hidden/tracker. .js URLs are emitted internal and only reach output modules if promoted for graph-importance, which races the end of the scan. That assertion now checks the unpacked body directly. 25/25 clean after.

[liquidsec]

@singlerider need re-review

[singlerider]

Re-reviewed at 6c28802. Both nits from the dismissed review are addressed, and the regex fix goes further than what I asked for. js_unpacker suite passes, 2 tests, three consecutive runs green.

🟢 Regex offloading

import re to import regex as re is the load-bearing half and I had not thought it through. A thread pool alone offloads nothing when stdlib re holds the GIL for the whole match, so wrapping the old module in run_in_executor_cpu would have moved the stall without fixing it. The split between helpers.re for single patterns and run_in_executor_cpu around the multi-step _decode / _deobfuscate / _extract_strings / _parse_source_map bodies is the right granularity: those do several passes each and batching them into one hop beats four round trips. RESOLVED ✅

🟢 Flaky dean-edwards assertion

Good catch, and the comment explains why the assertion moved rather than just moving it. Asserting on the unpacked body while leaving excavate's consumption covered by TestJsUnpackerBadsecretsChain keeps the coverage without depending on internal-event promotion. RESOLVED ✅

🔴 Blocking, _reemit_source still lands in --json output

You called this cosmetic and left it, but it is not confined to output rendering. HTTP_RESPONSE.sanitize_data does not strip it, so the key round-trips through event.json() and lands in the JSON/NDJSON output modules, verified locally. Anything reconstructing events from that output gets a key that only exists to steer _data_id, and the dedup identity of a reconstructed event now depends on a field that was never meant to be part of the event's public data. Strip underscore-prefixed keys in sanitize_data, or move the marker off data entirely and onto the event object.

[liquidsec]

@singlerider I disagree that this is a problem. Having the value in the JSON is an advantage, not a disadvantage: it says more about how the event was discovered, and more data in the output is better than less.

Also worth noting sanitize_data runs at event creation, not on output. Stripping the key there takes it off the event itself, and event.id hashes method|url|_reemit_source, so the unpacked response would come out with the same id as the original and get deduped away.

[singlerider]

@liquidsec

Stripping the key there takes it off the event itself, and event.id hashes method|url|_reemit_source

Third option: keep it off data, hang it on the event, surface it top-level in json() and restore it in event_from_json, same as archive_url at base.py:958; ran it end to end and ids, hashes, and round-trip identity all hold with data_json clean.

🔴 Blocking: please do it that way, the existing archive_url precedent settles it.

[liquidsec]

@singlerider I think the fundamental disagreement here is that there is some disadvantage to it being in json output itself. I prefer it there, its important context. When retracing steps using JSON data, it is very important to know that the source was a re-emittal.

and lands in the JSON/NDJSON output modules

i consider this a clear positive, not a negative