Skip to content

feat(attack-sim): attack-simulation benchmark harness (Phase A)#33

Merged
rennf93 merged 7 commits into
masterfrom
feat/attack-sim-harness
Jun 26, 2026
Merged

feat(attack-sim): attack-simulation benchmark harness (Phase A)#33
rennf93 merged 7 commits into
masterfrom
feat/attack-sim-harness

Conversation

@rennf93

@rennf93 rennf93 commented Jun 24, 2026

Copy link
Copy Markdown
Owner

What this adds

A self-contained attack-simulation benchmark under tests/attack_simulation/ (not shipped in the wheel) that measures the regex/detection layer with numbers:

  • Hybrid corpus — project-authored seed payloads (8 attack classes) + a deterministic mutation engine (12 evasion transforms + curated chains) + a benign corpus that probes false positives.
  • Drives the real SusPatternsManager.detect() through a production-shaped SecurityConfig (normalization active), via a singleton-safe context manager.
  • Metrics: detection rate, false-positive rate, F-score, per-attack-class recall, per-technique evasion-resistance matrix, and per-benign-category FP breakdown.
  • Two faces: make attack-sim emits report.json + report.md; a committed baseline.json + test_attack_simulation.py enforce no-regression (ratchet, not absolute thresholds).

How to run

make attack-sim                      # human + machine report into tests/attack_simulation/out/
uv run pytest tests/attack_simulation/   # 16 tests incl. the ratchet gate

Baseline this run records

detection 0.658 · FP 0.125 · F 0.791 · 4.64s / 320 scans.

The first run already surfaced real findings (this is the point — the harness measures, it does not fix):

  • Coverage holes (false negatives): SSTI {{7*7}}/${7*7} → 0.0, NoSQL {"$gt":""} → 0.0, boolean SQLi 1' OR '1'='1 → 0.5. These canonical payloads are not detected.
  • Chained-evasion bypass: unicode_homograph + url_encode collapses recall 0.688 → 0.188 (a normalization-order gap).
  • False positives concentrate in prose_sql_keywords (0.667) — SELECT…FROM in natural language; every other benign category is 0.0.

These become the measured, prioritized worklist for the next phase.

Scope / safety

Test-only additions plus an additive Makefile target and one .gitignore line. No shipped guard_core/ code changed.

Note: the pre-existing sync-mirror --check may report red on CI; it is unrelated to this change (the harness is test-only and does not touch the unasync sync mirror).

@github-actions github-actions Bot added tests Test suite changes build Makefile / Docker / setup.py / packaging labels Jun 24, 2026
@rennf93 rennf93 self-assigned this Jun 25, 2026
@rennf93
rennf93 merged commit 7ce43c7 into master Jun 26, 2026
10 of 11 checks passed
@rennf93
rennf93 deleted the feat/attack-sim-harness branch June 26, 2026 01:24
rennf93 added a commit that referenced this pull request Jul 1, 2026
…-mirror repair (#38)

Assembles the 3.3.0 release from the detection work merged since v3.2.0 (#33-#37) plus the ipinfo-None deprecation fix from the 3.2.1 line.

- ipinfo_token/ipinfo_db_path DeprecationWarning no longer fires when the field is explicitly None (spurious warning for callers forwarding an optional setting).

- Declare SusPatternsManager._threat_score_threshold and coerce _regex_anomaly to float, fixing the mypy attr-defined / no-any-return errors introduced by #36.

- Exclude the async-only attack_simulation harness from unasync mirroring (its data corpus is not Python-transformable) and regenerate the previously-missing sync test mirrors; check-sync is green.

- Bump version to 3.3.0 and add CHANGELOG / release-notes entries.

Co-authored-by: Renn F <rennf93@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build Makefile / Docker / setup.py / packaging tests Test suite changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant