Add nowafpls Cloudflare WAF bypass detector

[liquidsec]

Summary

Adds a WAF-bypass technique based on prepending a large junk padding to POST bodies —
inspired by the nowafpls burp extension.
Cloudflare (and several other inline WAFs) has a bounded inspection buffer; once
the request body grows past it, the malicious payload lands after the inspection
window and reaches the app unfiltered. This branch delivers:

• bbot/core/helpers/nowafpls.py — shared helper. is_bypassable(event) probes
  the host once (baseline vs unpadded-malicious vs padded-malicious via
  http_compare), memoized per host. pad_form_body(event, body) and
  pad_json(event, data) are drop-ins for adversarial POST bodies.
• bbot/modules/nowafpls.py — dedicated detector that emits a FINDING when a
  WAF-tagged host is bypassable, with the padding size in the description.
• Lightfuzz integration — tri-state avoid_wafs config (always / never /
  try_bypasses, default try_bypasses). Booleans are still accepted (true ->
  always, false -> never), so configs written against the previous boolean
  option keep working. Submodule prepare_request transparently
  pads POSTPARAM / BODYJSON probes when the host's WAF is bypassable, and flips
  used_nowafpls=True so lightfuzz tags the finding used-nowafpls.
• ajaxpro + generic_ssrf — POST paths now call the padding helpers so their
  fuzz traffic uses the same bypass on WAF-tagged hosts.
• lightfuzz-xss preset — enables try_get_as_post so GET params get re-fuzzed
  as padded POSTs, catching XSS reachable from an attacker-controlled form when the
  WAF blocks the raw payload.

Shared-framework change: _is_http_wildcard_host

BaseModule._is_http_wildcard_host no longer returns True as soon as the host is
flagged a wildcard responder. It now probes the specific URL against the wildcard
baseline and reports a wildcard only when the response matches it, so a URL that
diverges from the baseline is treated as a real endpoint and is no longer skipped.

For WEB_PARAMETER GETPARAMs the probe bakes the parameter into the URL, since a bare
path can be a catchall while ?name=value is a real endpoint.

This is shared framework behavior and affects every module that relies on wildcard-host
filtering, not just lightfuzz. Covered by
test_base_module_is_http_wildcard_host_per_url in bbot/test/test_step_1/test_web.py.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs nowafpls (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	2.18ms	2.15ms	-1.5% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	9.29ms	9.28ms	-0.1% ⚪	✅
Large Closest Match Lookup	216.01ms	243.21ms	+12.6% 🟡🟡	⚠️
Realistic Closest Match Workload	122.48ms	123.51ms	+0.8% ⚪	✅
Event Memory Medium Scan	1402 B/event	1405 B/event	+0.3% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	256.41ms	261.87ms	+2.1% ⚪	✅
Event Validation Full Scan Startup Large Batch	337.25ms	347.41ms	+3.0% ⚪	✅
Make Event Autodetection Small	14.91ms	15.53ms	+4.2% ⚪	✅
Make Event Autodetection Large	151.06ms	155.60ms	+3.0% ⚪	✅
Make Event Explicit Types	6.28ms	6.63ms	+5.6% ⚪	✅
Excavate Single Thread Small	3.009s	3.003s	-0.2% ⚪	✅
Excavate Single Thread Large	6.488s	6.767s	+4.3% ⚪	✅
Excavate Parallel Tasks Small	3.213s	3.176s	-1.1% ⚪	✅
Excavate Parallel Tasks Large	4.702s	4.646s	-1.2% ⚪	✅
Intercept Throughput Small	697.89ms	702.13ms	+0.6% ⚪	✅
Intercept Throughput Medium	685.75ms	728.78ms	+6.3% ⚪	✅
Dns Throughput Quiet	1.851s	1.966s	+6.2% ⚪	✅
Dns Throughput Loaded	1.252s	1.282s	+2.4% ⚪	✅
Dns Throughput Inherited	1.667s	1.687s	+1.2% ⚪	✅
Is Ip Performance	1.34ms	1.40ms	+4.1% ⚪	✅
Make Ip Type Performance	118.31µs	124.28µs	+5.0% ⚪	✅
Mixed Ip Operations	1.42ms	1.48ms	+4.0% ⚪	✅
Memory Use Web Crawl	1.0 MB	1.2 MB	+15.8% 🟡🟡	⚠️
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	12.7 MB	11.2 MB	-11.5% 🟢🟢	🚀
Scan Throughput 100	2.320s	2.133s	-8.0% ⚪	✅
Scan Throughput 1000	15.584s	15.817s	+1.5% ⚪	✅
Typical Queue Shuffle	3.50µs	3.53µs	+0.9% ⚪	✅
Priority Queue Shuffle	16.70µs	17.25µs	+3.3% ⚪	✅

🎯 Performance Summary

+ 1 improvement 🚀
! 2 regressions ⚠️
  28 unchanged ✅

🔍 Significant Changes (>10%)

• Large Closest Match Lookup: 12.6% 🐌 slower
• Memory Use Web Crawl: 15.8% 🐌 more memory
• Memory Use Parallel Chains: 11.5% 🚀 less memory

---

🐍 Python Version 3.11.16

[codecov]

Codecov Report

❌ Patch coverage is 94.83037% with 32 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (a394d12) to head (e898ab4).
⚠️ Report is 9 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/core/helpers/nowafpls.py	88%	12 Missing ⚠️
...t/test_step_2/module_tests/test_module_nowafpls.py	94%	6 Missing ⚠️
...st_step_2/module_tests/test_module_generic_ssrf.py	94%	4 Missing ⚠️
bbot/modules/lightfuzz/lightfuzz.py	93%	3 Missing ⚠️
bbot/core/helpers/web/web.py	50%	2 Missing ⚠️
...st/test_step_2/module_tests/test_module_ajaxpro.py	96%	2 Missing ⚠️
.../test_step_2/module_tests/test_module_lightfuzz.py	99%	2 Missing ⚠️
bbot/modules/nowafpls.py	97%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #3336    +/-   ##
======================================
+ Coverage     90%     90%    +1%     
======================================
  Files        454     457     +3     
  Lines      47081   47667   +586     
======================================
+ Hits       42320   42879   +559     
- Misses      4761    4788    +27     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[singlerider]

Tests pass (nowafpls/ajaxpro/generic_ssrf, test_web, and the full lightfuzz suite). The prepare_request/compare_baseline sync-to-async conversion touches every submodule; verified every call site is awaited, no un-awaited coroutines.

Two things to flag before merge:

• Breaking config change: avoid_wafs went from bool to Literal[always,never,try_bypasses]. An existing avoid_wafs: true/false in a user config is now rejected at load time. In-repo presets are migrated but external configs will break. Recommend a coercing validator (True->always, False->never) or a breaking-change changelog note.

• _is_http_wildcard_host in base.py was rewritten to probe the specific URL (baking in GETPARAM values) against the wildcard baseline instead of returning True. It is well tested via the new test_web cases, but it is a shared-framework behavior change with no mention in the PR body. Please call it out.

Nit: is_bypassable memoizes per-host but ignores padding_size/payload on cache hits, so if the standalone module and lightfuzz probe the same host, the first caller's params win.

[liquidsec]

Both addressed.

avoid_wafs now accepts the boolean form (true -> always, false -> never) via a before-validator, so existing configs load unchanged. Worth noting schema validation doesn't rewrite the runtime config, so the module applies the same coercion where it reads the option; the validator alone would have let true load and then behave as never. Both halves are tested in test_validate_preset.py. webbrute.avoid_wafs stays a bool on purpose, we don't want to brute force dirs through a WAF just because nowafpls exists.

Called out the _is_http_wildcard_host change in the PR body. Also merged dev.

[liquidsec]

Three defects were causing WAF-tagged hosts to be skipped entirely, one of which
could also produce a false positive. Fixed in 0d98eb6.

1. no_interference was treated the same as blocked

lightfuzz.filter_event gated on not result.bypassed under
avoid_wafs: try_bypasses. BypassResult.bypassed is only true for
STATUS_BYPASSED, so that single check collapsed three distinct verdicts into
one rejection:

• no_interference - the payload reached the application untouched
• blocked - the gate held and padding did not help
• error - the probe never produced a verdict

Only the last two are reasons to skip. no_interference means there is nothing
to work around, so those events should be fuzzed normally. Since the helper
memoizes per host, the first probe's verdict applied to every subsequent event
on that host, so a host fronted by a CDN that isn't actively blocking received
no fuzzing at all.

This matters because the waf tag is assigned from the CDN/WAF provider's
identity, not from observed blocking. A host can carry the tag while gating
nothing. nowafpls exists to measure what the host actually does, so the
measurement should win over the tag.

2. _post_capable was gating the probe rather than the bypass

The same branch rejected any event that could not fire a POST probe before
consulting nowafpls at all:

if not self._post_capable(event):
    return False, "WAF-tagged event has no POST-style probe to pad"
result = await self.helpers.nowafpls.is_bypassable(event)

_post_capable is false for every URL event, and since try_get_as_post
defaults to false, for every GETPARAM as well. On a tagged host the majority of
discovered parameters were therefore dropped before any measurement happened.
Padding is body-only, so that gate is only meaningful once the verdict is
bypassed; when nothing is gating the payload there is no bypass to apply and
no reason to restrict which event types get fuzzed.

3. HttpCompare.compare() reported a dead request as a baseline match

if subject_response is None:
    return (True, "403", reflection, subject_response)

A timeout or connection reset produced match=True. For header and cookie
fuzzing that is a reasonable conservative default, but two other consumers read
it as a positive verdict:

• nowafpls - a killed unpadded request became "matches baseline", so the
  probe reported no_interference and returned before ever testing padding. A
  WAF that drops the connection instead of serving a block page was invisible.
  The mirror case is worse: a killed padded request became bypassed,
  emitting a CONFIRMED finding on no evidence.
• HTTP wildcard detection (web.py, base.py) - a dead probe became "root
  matches the random-path baseline", so the host was declared a wildcard
  responder. lightfuzz.filter_event checks this before any WAF logic, so a
  single timeout could black out a host completely.

The reasons value on that path was also a bare string while every other path
returns a list, so consumers doing ",".join(reasons) rendered it as 4,0,3.

Changes

• compare() takes none_is_match (default True, preserving existing
  behavior) and returns ["request_failed"] as a list. nowafpls and both
  wildcard probes pass False; the wildcard paths now map a dead probe to
  "unknown" rather than to a verdict.
• filter_event branches on the explicit status and applies _post_capable
  only when the verdict is bypassed.
• Probe verdicts are logged through a shared BypassResult.summary carrying the
  error and diff reasons, reported at INFO. Previously the only report of a
  verdict was a debug line, and nothing was logged at all when lightfuzz
  consulted the helper without the nowafpls module enabled.

Tests

Each new test was confirmed to fail against the pre-fix code:

• Test_Lightfuzz_filter_event_try_bypasses - accept/reject across all four
  statuses for URL, GETPARAM and POSTPARAM events.
• Test_Nowafpls_try_bypasses_no_interference - a host with no interference is
  fuzzed, and unpadded.
• TestNowafplsUnpaddedConnectionKilled - a killed unpadded request with a
  succeeding padded one is a bypass, not silence.
• TestNowafplsPaddedConnectionKilled - a killed padded request is not a bypass.
• test_base_module_is_http_wildcard_host_per_url gains a dead-probe case. Its
  FakeCompare stub previously returned None for the response on every call,
  which only happens for real when the request fails; it now returns a response
  object and models the failure explicitly.

180 tests pass across lightfuzz, nowafpls, test_web, paramminer
(headers/getparams/cookies), bypass403, url_manipulation and webbrute.

[liquidsec]

@singlerider looks like this may have inadvertently also fixed a bug that was in stable currently. So its a high priority now. @shart123456

[singlerider]

Re-reviewed at cb8db4f. Both flagged items resolved, and 0d98eb6 is a real fix on top. Suites pass on the branch: lightfuzz 125, nowafpls + validate_preset 36, test_web + bypass403 + paramminer_headers 29.

🟢 avoid_wafs backward compatibility

RESOLVED ✅. Your point about the validator not rewriting the runtime config is the part I would have missed: a before-validator alone would have let true load and then behave as never, since lightfuzz.setup() reads the raw dict. Coercing in both places is correct, and test_lightfuzz_avoid_wafs_bool_coerced_at_runtime pins the half that actually decides behavior. Confirmed "bogus" and 5 still fail at load with the expected-one-of message.

🟢 _is_http_wildcard_host documented

RESOLVED ✅. The PR body now has its own section calling it out as shared framework behavior with the GETPARAM rationale and the covering test named.

🟢 none_is_match on a dead probe

Not something I caught on the first pass and it is the strongest commit in the branch. compare() returning (True, "403", ...) meant a killed request read as a baseline match, so nowafpls scored a dropped unpadded request as no-interference and a dropped padded one as a bypass, which is exactly backwards for the thing this PR exists to detect. Defaulting to True keeps every other call site unchanged, and the three that care opt out explicitly. Turning reasons into a list also fixes the "4,0,3" render in paramminer's ",".join(reasons).

🟢 _post_capable ordering

Applying it before the probe meant a WAF-tagged host dropped every URL and GETPARAM event regardless of the verdict. Gating it on STATUS_BYPASSED is right, since the padding is what needs a body in the first place.

🔴 Blocking, STATUS_ERROR now silently skips the event

filter_event lumps STATUS_ERROR in with STATUS_BLOCKED and returns a bare False, so a probe that failed for an infrastructure reason (a baseline that could not be established, a transient timeout) is indistinguishable from a WAF that held. The old code at least had one meaning. Either return the string form so the skip is reported, or treat an error as unknown and fall through to fuzzing, but a bare False on an error path loses the event with no record outside debug logging.

[liquidsec]

Addressed in 58bb592.

STATUS_ERROR no longer skips silently

filter_event branches on the two statuses separately now, and both carry a reason:

• blocked -> "WAF blocked the payload and body padding did not help (status=blocked provider=... diff_reasons=...)"
• error -> "WAF bypass probe was inconclusive (status=error provider=... error=could not baseline ...)"

The hand-rolled parsed_url / self.debug block is gone, since _event_postcheck_inner already appends the returned reason to the rejection message.

Two things I decided against, with the reasoning:

An error still skips rather than falling through to fuzzing. STATUS_ERROR can only come from HttpCompareError, which is raised only by HttpCompare._baseline, so the verdict means the host's baseline could not be established. Every lightfuzz submodule baselines through that same code, so falling through would re-fail per submodule per event without producing anything.

Error verdicts stay memoized. Dropping them from the per-host cache would re-probe on every event, and _baseline is two requests at retries=2 / timeout=10 plus a 0.5s sleep, so one unreachable host could stall filter_event for close to a minute per event. The reported reason is what makes the transient case visible instead of silent.

Verdict logging

is_bypassable logs the verdict once per host at verbose, keyed off whether the call created the probe task. Previously a lightfuzz-only run (nowafpls module not enabled) had no record of a verdict above debug, which is the other half of the "no record" problem.

Tests

Test_Lightfuzz_filter_event_try_bypasses now asserts that every rejection carries a reason, that blocked and error produce different reasons, and that the verdict is included in the string. Confirmed failing against the pre-fix code with AssertionError: URL rejection on status=blocked must carry a reason, got False.

Locally: lightfuzz 125 passed, test_web + nowafpls + test_presets 53 passed. Branch CI is still running as of this comment.