Rustification - BlastDiff

[liquidsec]

Rust reimplementation of BBOT's HTTP diffing engine, following the same pattern as BlastDNS (massdns replacement) and blasthttp (native HTTP engine). Ship as a Rust crate with Python bindings and swap it in behind the existing interface.

What it replaces

bbot/core/helpers/diff.py, specifically the HttpCompare class and its compare_body / header-diff logic, which is built on the Python DeepDiff library.

Consumers today:

• virtualhost (body similarity)
• waf_bypass (content fingerprint comparison)
• lightfuzz/crypto submodule (compare_body on probe responses)

Why

DeepDiff is the bottleneck. Known problems already tracked:

• HTTP wildcard detection can stall an entire scan for tens of minutes in compare_body (quadratic DeepDiff pairing, holds the GIL) #3339: HTTP wildcard detection stalls a scan for tens of minutes in compare_body (quadratic DeepDiff pairing, holds the GIL)
• Selectively Replace DeepDiff Due to Performance Issues #3340: selectively replace DeepDiff due to performance issues

The work is CPU-bound and currently holds the GIL, which is exactly what the Blast family exists to fix.

Scope

• Rust crate implementing body and header comparison with the same semantics HttpCompare relies on (duplicate-header handling, JSON/XML awareness, order sensitivity where it matters).
• Python bindings exposing a drop-in for compare_body and the header diff.
• Wire into diff.py behind the current HttpCompare API so callers do not change.

Acceptance criteria

• compare_body and header diff produce equivalent results to the current DeepDiff path on the existing diff tests.
• Wildcard detection on the HTTP wildcard detection can stall an entire scan for tens of minutes in compare_body (quadratic DeepDiff pairing, holds the GIL) #3339 repro no longer stalls; runs in seconds, not minutes.
• Comparison work no longer holds the GIL.
• Existing consumers (virtualhost, waf_bypass, lightfuzz) pass unchanged.

Open

• Whether scope is just compare_body or the full HttpCompare class.
• Keep DeepDiff as a fallback during rollout vs hard cutover.