Fix unbounded lightfuzz queue growth from parameter re-discovery loop

[liquidsec]

Problem

lightfuzz emits a baseline HTTP_RESPONSE for each parameter it probes. excavate mines those responses and emits the parameters it finds, which come back to lightfuzz, which probes them and emits more baselines. The loop only terminates if dedup collapses the repeats.

WEB_PARAMETER._data_id keyed on the full URL including its query string. On pages that vary their form on every load (rotating CSRF tokens, randomly named honeypot fields), each generation produced a unique key, so dedup never fired and lightfuzz's incoming queue grew without bound.

The same failure happens when the varying part is in the path rather than the query string, for example a session id appended to the path on every redirect.

Changes

Three independent fixes, one per commit.

1. Dedupe WEB_PARAMETER on the page, not the full URL. A parameter is identified by its name and type, both already separate fields in the dedup key, so the query string of whichever request revealed it is context rather than identity. This also makes POSTPARAM and COOKIE consistent with GETPARAM, whose query string excavate already strips.

url_querystring_collapse: False still opts back in to treating sibling parameter values as significant, so lightfuzz-max and lightfuzz-xss keep fuzzing each variant separately. Those keys are now normalized, so parameter ordering no longer affects dedup.

2. Cap lightfuzz baseline response generations. New max_baseline_generations (default 10). lightfuzz stops emitting baseline responses once a parameter is that many lightfuzz generations deep, starving excavate of new material. Applied at emit time rather than in filter_event, which runs after dequeue and so would not have prevented the queue from growing.

This is what bounds the loop where dedup cannot, which is the url_querystring_collapse: False case above. Legitimate post-submit discovery runs a few generations deep, so 10 leaves ample headroom.

3. Reject URLs whose path repeats a path parameter. New url_max_path_param_repeats (default 10), enforced at ingress via the existing blacklisted tag so the event never reaches a module queue. The rule is structural, keyed on repetition rather than a list of known session parameter names.

Only ;key=value path parameters count. Ordinary path segments deliberately do not: a deep path that repeats a directory name is finite, and depth is already bounded by web_spider_depth. Set to 0 to disable.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs lightfuzz-queue-amplification (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	2.55ms	2.60ms	+2.0% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	11.44ms	11.10ms	-3.0% ⚪	✅
Large Closest Match Lookup	270.29ms	276.78ms	+2.4% ⚪	✅
Realistic Closest Match Workload	151.95ms	144.46ms	-4.9% ⚪	✅
Event Memory Medium Scan	1403 B/event	1402 B/event	-0.1% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	298.19ms	318.69ms	+6.9% ⚪	✅
Event Validation Full Scan Startup Large Batch	393.42ms	403.35ms	+2.5% ⚪	✅
Make Event Autodetection Small	17.99ms	18.23ms	+1.4% ⚪	✅
Make Event Autodetection Large	181.16ms	183.12ms	+1.1% ⚪	✅
Make Event Explicit Types	7.63ms	7.87ms	+3.1% ⚪	✅
Excavate Single Thread Small	3.408s	3.413s	+0.2% ⚪	✅
Excavate Single Thread Large	9.523s	9.609s	+0.9% ⚪	✅
Excavate Parallel Tasks Small	3.632s	3.570s	-1.7% ⚪	✅
Excavate Parallel Tasks Large	5.603s	5.543s	-1.1% ⚪	✅
Intercept Throughput Small	784.95ms	743.55ms	-5.3% ⚪	✅
Intercept Throughput Medium	739.79ms	788.34ms	+6.6% ⚪	✅
Dns Throughput Quiet	2.162s	2.136s	-1.2% ⚪	✅
Dns Throughput Loaded	1.399s	1.554s	+11.1% 🟡🟡	⚠️
Dns Throughput Inherited	1.925s	1.876s	-2.5% ⚪	✅
Is Ip Performance	1.59ms	1.61ms	+1.4% ⚪	✅
Make Ip Type Performance	134.38µs	135.11µs	+0.5% ⚪	✅
Mixed Ip Operations	1.67ms	1.67ms	-0.4% ⚪	✅
Memory Use Web Crawl	1.0 MB	1.0 MB	+0.0% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.2% ⚪	✅
Memory Use Parallel Chains	10.7 MB	11.6 MB	+8.4% ⚪	✅
Scan Throughput 100	2.489s	2.460s	-1.2% ⚪	✅
Scan Throughput 1000	17.484s	17.327s	-0.9% ⚪	✅
Typical Queue Shuffle	4.25µs	4.22µs	-0.6% ⚪	✅
Priority Queue Shuffle	19.78µs	19.64µs	-0.7% ⚪	✅

🎯 Performance Summary

! 1 regression ⚠️
  30 unchanged ✅

🔍 Significant Changes (>10%)

• Dns Throughput Loaded: 11.1% 🐌 slower

---

🐍 Python Version 3.11.16

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90%. Comparing base (a394d12) to head (7029c11).

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #3405    +/-   ##
======================================
+ Coverage     90%     90%    +1%     
======================================
  Files        454     454            
  Lines      47081   47229   +148     
======================================
+ Hits       42320   42479   +159     
+ Misses      4761    4750    -11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.