Skip to content

restore cloudcheck tag propagation to URL/OPEN_TCP_PORT children#3309

Merged
liquidsec merged 6 commits into
devfrom
cloudcheck-cloud-tag-fixes
Jul 17, 2026
Merged

restore cloudcheck tag propagation to URL/OPEN_TCP_PORT children#3309
liquidsec merged 6 commits into
devfrom
cloudcheck-cloud-tag-fixes

Conversation

@liquidsec

Copy link
Copy Markdown
Collaborator

Summary

_minimize() was wiping _resolved_hosts on parent events once their consumers finished. That starved dnsresolve line 156 (event._resolved_hosts = main_host_event.resolved_hosts) when child URL / OPEN_TCP_PORT events later reached it, and without the parent's IPs on the child, cloudcheck's per-host inheritance filter had nothing to match against. Cloud tags were never propagating past the DNS_NAME.

Fix: stop wiping _resolved_hosts in _minimize(). It is a small frozenset that the memory optimization was not really targeting.

Also updates waf_bypass and virtualhost, which still filtered on the pre-f561bda1e composed tags (cdn-cloudflare, waf-cloudflare, cdn-imperva, cdn-akamai, cdn-cloudfront). They now key off the flat provider name that cloudcheck emits.

Regression tests

  • test_module_cloudcheck.py: _minimize() preserves resolved_hosts; URL and OPEN_TCP_PORT children inherit cloud tags.
  • test_module_waf_bypass.py: both cloudflare and imperva tags trigger protected_domains detection; untagged URLs don't.
  • test_module_virtualhost.py: each of cloudflare, imperva, akamai, cloudfront causes filter_event to skip; untagged URLs still pass.

`_minimize()` wiped `_resolved_hosts` on the parent DNS_NAME once its consumers
finished, which starved `dnsresolve` line 156 when child URL / OPEN_TCP_PORT
events later reached it. Without the parent's IPs on the child, cloudcheck's
per-host inheritance filter had nothing to match against and the cloud tags
never propagated. Stop wiping `_resolved_hosts` in `_minimize()`: it is a
small frozenset that the memory optimization was not really targeting.

Also updates the modules still checking the pre-f561bda1e composed tags
(`cdn-cloudflare`, `waf-cloudflare`, `cdn-imperva`, `cdn-akamai`,
`cdn-cloudfront`) to key off the flat provider name that cloudcheck now emits.

Regression tests cover both sides: cloudcheck asserts `_minimize()` preserves
resolved_hosts and that URL / OPEN_TCP_PORT children inherit cloud tags;
waf_bypass asserts both cloudflare and imperva tags trigger protection
detection; virtualhost asserts each of the four flat provider names causes
filter_event to skip the URL.
Comment thread bbot/test/test_step_2/module_tests/test_module_cloudcheck.py Dismissed
Comment thread bbot/test/test_step_2/module_tests/test_module_cloudcheck.py Dismissed
Comment thread bbot/test/test_step_2/module_tests/test_module_cloudcheck.py Dismissed
@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing dev (baseline) vs cloudcheck-cloud-tag-fixes (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 4.26ms 4.32ms +1.4%
Bloom Filter Large Scale Dns Brute Force 18.65ms 17.88ms -4.1%
Large Closest Match Lookup 367.67ms 361.75ms -1.6%
Realistic Closest Match Workload 193.79ms 197.98ms +2.2%
Event Memory Medium Scan 1402 B/event 1402 B/event +0.0%
Event Memory Large Scan 1527 B/event 1527 B/event +0.0%
Event Validation Full Scan Startup Small Batch 411.68ms 409.89ms -0.4%
Event Validation Full Scan Startup Large Batch 571.58ms 540.64ms -5.4%
Make Event Autodetection Small 26.04ms 25.89ms -0.6%
Make Event Autodetection Large 265.46ms 262.66ms -1.1%
Make Event Explicit Types 11.51ms 11.61ms +0.8%
Excavate Single Thread Small 4.531s 4.251s -6.2%
Excavate Single Thread Large 10.536s 9.953s -5.5%
Excavate Parallel Tasks Small 4.701s 4.349s -7.5%
Excavate Parallel Tasks Large 6.850s 6.454s -5.8%
Intercept Throughput Small 890.38ms 892.36ms +0.2%
Intercept Throughput Medium 883.11ms 877.66ms -0.6%
Dns Throughput Quiet 2.878s 2.822s -1.9%
Dns Throughput Loaded 1.859s 1.821s -2.0%
Dns Throughput Inherited 2.642s 2.505s -5.2%
Is Ip Performance 2.31ms 2.33ms +0.8%
Make Ip Type Performance 241.16µs 235.13µs -2.5%
Mixed Ip Operations 2.42ms 2.40ms -1.1%
Memory Use Web Crawl 1.0 MB 1.0 MB +0.0%
Memory Use Subdomain Enum 28.3 MB 28.3 MB +0.0%
Memory Use Deep Chain 5.0 MB 5.0 MB +0.0%
Memory Use Parallel Chains 11.9 MB 12.2 MB +2.2%
Scan Throughput 100 2.860s 2.897s +1.3%
Scan Throughput 1000 21.132s 20.726s -1.9%
Typical Queue Shuffle 5.46µs 5.36µs -1.7%
Priority Queue Shuffle 26.12µs 26.30µs +0.7%

🎯 Performance Summary

No significant performance changes detected (all changes <10%)


🐍 Python Version 3.11.15

@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.43750% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (6f62e97) to head (8903230).
⚠️ Report is 7 commits behind head on dev.

Files with missing lines Patch % Lines
bbot/modules/waf_bypass.py 89% 1 Missing ⚠️
Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #3309   +/-   ##
=====================================
+ Coverage     90%     90%   +1%     
=====================================
  Files        450     450           
  Lines      46229   46285   +56     
=====================================
+ Hits       41484   41547   +63     
+ Misses      4745    4738    -7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@en0f
en0f self-requested a review July 16, 2026 19:02

@en0f en0f left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reran tests. All set.

@liquidsec
liquidsec merged commit 1e01051 into dev Jul 17, 2026
26 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants