feat(attack-sim): attack-simulation benchmark harness (Phase A)#33
Merged
Conversation
…ategory breakdowns
rennf93
added a commit
that referenced
this pull request
Jul 1, 2026
…-mirror repair (#38) Assembles the 3.3.0 release from the detection work merged since v3.2.0 (#33-#37) plus the ipinfo-None deprecation fix from the 3.2.1 line. - ipinfo_token/ipinfo_db_path DeprecationWarning no longer fires when the field is explicitly None (spurious warning for callers forwarding an optional setting). - Declare SusPatternsManager._threat_score_threshold and coerce _regex_anomaly to float, fixing the mypy attr-defined / no-any-return errors introduced by #36. - Exclude the async-only attack_simulation harness from unasync mirroring (its data corpus is not Python-transformable) and regenerate the previously-missing sync test mirrors; check-sync is green. - Bump version to 3.3.0 and add CHANGELOG / release-notes entries. Co-authored-by: Renn F <rennf93@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A self-contained attack-simulation benchmark under
tests/attack_simulation/(not shipped in the wheel) that measures the regex/detection layer with numbers:SusPatternsManager.detect()through a production-shapedSecurityConfig(normalization active), via a singleton-safe context manager.make attack-simemitsreport.json+report.md; a committedbaseline.json+test_attack_simulation.pyenforce no-regression (ratchet, not absolute thresholds).How to run
Baseline this run records
detection 0.658 · FP 0.125 · F 0.791 · 4.64s / 320 scans.
The first run already surfaced real findings (this is the point — the harness measures, it does not fix):
{{7*7}}/${7*7}→ 0.0, NoSQL{"$gt":""}→ 0.0, boolean SQLi1' OR '1'='1→ 0.5. These canonical payloads are not detected.unicode_homograph + url_encodecollapses recall 0.688 → 0.188 (a normalization-order gap).prose_sql_keywords(0.667) —SELECT…FROMin natural language; every other benign category is 0.0.These become the measured, prioritized worklist for the next phase.
Scope / safety
Test-only additions plus an additive
Makefiletarget and one.gitignoreline. No shippedguard_core/code changed.Note: the pre-existing
sync-mirror --checkmay report red on CI; it is unrelated to this change (the harness is test-only and does not touch the unasync sync mirror).