loop iter 4: fix-impact gate signal + saturation checkpoint

- build_security_dataset.py: add title-level fix-impact to the admission GATE
  (completes the independent-signal set); description-level rejected as it fires
  on release notes that merely list a "fix crash". Net +1 new admit — title
  crash fixes already score >=0.5, confirming deterministic recall is saturated.
- test_security_dataset.py: gate test now covers the fix-impact signal.
- IMPROVEMENT_LOG: loop-until-dry termination met for deterministic signals;
  next lever is clone-based backport mining (A1) or the deferred LLM pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: grandchildrice

collection/IMPROVEMENT_LOG.md

@@ -26,11 +26,23 @@ fuzzer report, [A1] cherry-pick/backport. Architecture: **multi-signal scoring**
 | 2 | [C] T2b NVD substring-match FP filter (+ source fix in crawl_cve.py) | 885 | 1877 | 173 / 3 | **Authoritative tier was 17% garbage** — "geth" matched as substring in gethostbyaddr/GetHost/"Gether Technology"/Linux usb. Dropped 49 unrelated CVEs (glibc/X.Org/Samba/kernel); 6 real client CVEs kept. Precision iteration: A-tier now clean. |
 | 3 | [C5'] fix-verb × impact co-occurrence signal | **1367** | 1877 | 173 / 3 | **C5-via-API dead end**: client PRs almost never use formal `Closes #N` (0/116 inline-ref PRs had closingIssuesReferences). Pivoted to the offline reservoir: "fix panic on…"/"prevent race in…" adjacency is a strong independent defect signal. +482 crash/DoS fixes promoted C→B. 907 rows now ≥2 signals (was 376). `enrich_linked_issues.py` kept as a tool (with `--require-inline-ref`). |
 
-**Loop status:** deterministic precision+recall largely worked through. Essential
-slice 173 → 1367 (7.9×) across 3 iterations. Remaining levers are heavier:
-[A1] cherry-pick/backport (needs local clones), [B4] review-comment mining
-(per-PR API), and the deferred LLM STRIDE/CWE classification (biggest recall
-lever — would admit the ~16k unrated stealth fixes; user deferred it, "ひとまずskip").
+| 4 | fix-impact added to the GATE (title-level, recall) + test coverage | 1367 | 1878 | 173 / 3 | Title-level fix-impact admits only **+1** new row — title crash fixes already score ≥0.5. Description-level would admit 200 but pulls release-note noise (rejected). **Deterministic recall is saturated.** |
+
+**Loop status — deterministic techniques exhausted (loop-until-dry hit).**
+Recall was flat across all 4 iterations (corpus already ~complete at 18,475 raw
+/ 1,878 curated); the wins were **precision + tiering**: essential slice
+173 → 1367 (7.9×), authoritative tier de-garbaged (−49 unrelated CVEs), −1417
+CI/docs/dep-bump noise. Two consecutive iterations (3,4) added <10 new corpus
+rows → termination condition met for deterministic signals.
+
+**Remaining levers require a technique switch (heavier / user-gated):**
+- [A1] cherry-pick/backport via local clones — the one untapped *new-source*
+  (geth silently backports fixes to release branches). Uncertain yield, ~1 GB
+  clones. **Next iteration to attempt.**
+- [B4] review-comment mining — per-PR API, slow, low expected yield (like C5).
+- **LLM STRIDE/CWE classification** — the biggest recall lever by far (admits the
+  ~16k unrated stealth fixes the keyword gate drops), but the user deferred it
+  ("ひとまずskip"). Re-enabling it is the real path to *comprehensive* coverage.
 
 ### Next iterations (API-heavy hidden-fix signals — recall expansion)
 - **iter 2** [C5] linked-issue / fuzzer-report signal: `gh pr view --json closingIssuesReferences`, score issue body for crash/panic/fuzz + reporter (oss-fuzz/Guido Vranken).

data/ethereum_vulns.parquet

@@ -1,6 +1,6 @@
 {
   "domain": "ethereum",
-  "n_rows": 1877,
+  "n_rows": 1878,
   "schema": [
     "id",
     "source_platform",
@@ -31,19 +31,20 @@
     "t2_noise_dropped": 1417,
     "t2b_nvd_fp_dropped": 49,
     "after_t1": 16998,
-    "security_rows": 1877,
-    "low_signal_dropped": 15121,
+    "security_rows": 1878,
+    "low_signal_dropped": 15120,
     "by_confidence": {
       "medium": 1544,
-      "high": 333
+      "high": 333,
+      "low": 1
     },
     "by_authority_tier": {
       "B_corroborated": 1134,
-      "C_candidate": 510,
+      "C_candidate": 511,
       "A_authoritative": 233
     },
     "by_n_signals": {
-      "1": 555,
+      "1": 556,
       "2": 907,
       "3": 345,
       "4": 65,
@@ -58,21 +59,21 @@
       "reth": 161,
       "prysm": 114,
       "nethermind": 104,
-      "teku": 90,
+      "teku": 91,
       "besu": 89,
       "grandine": 16,
       "consensus-specs": 2
     },
     "by_severity": {
-      "Unrated": 963,
+      "Unrated": 964,
       "Info": 773,
       "High": 63,
       "Medium": 54,
       "Low": 21,
       "Critical": 3
     },
     "by_score": {
-      "0.0": 34,
+      "0.0": 35,
       "0.3": 4,
       "0.5": 1208,
       "0.8": 298,

data/manifest.json

@@ -322,12 +322,19 @@ def build(df: pd.DataFrame) -> tuple[pd.DataFrame, dict]:
     ]
 
     # GATE — union of independent signals
-    has_id = (df["title"].fillna("") + " " + df["description"].fillna("")).str.contains(CVE_RE)
+    gate_blob = df["title"].fillna("") + " " + df["description"].fillna("")
+    has_id = gate_blob.str.contains(CVE_RE)
     has_sev = df["severity"].fillna("").str.lower().isin(RATED_SEV)
     has_kw = df["security_score"] >= 0.5
     has_stride = ~df["stride"].fillna("Other").isin(["Other"])
     has_cwe = ~df["cwe_top25"].fillna("N/A").isin(["N/A"])
-    df["security_relevant"] = has_id | has_sev | has_kw | has_stride | has_cwe
+    # Recall expansion: a fix-verb × crash-class impact co-occurrence in the
+    # TITLE admits real crash/DoS fixes whose only keyword sat in the description
+    # (score 0.3, below the 0.5 threshold). Title-only — a description-level
+    # match also fires on release notes that merely *list* a "fix crash". T2
+    # already dropped dep-bump/CI titles upstream.
+    has_fiximpact = df["title"].fillna("").str.contains(FIX_IMPACT_RE)
+    df["security_relevant"] = has_id | has_sev | has_kw | has_stride | has_cwe | has_fiximpact
 
     sec = df[df["security_relevant"]].copy()
     sec["confidence"] = sec.apply(confidence_tier, axis=1)

pipeline/build_security_dataset.py

@@ -48,7 +48,17 @@ def test_every_row_has_a_security_signal(df):
     has_cwe = ~df["cwe_top25"].fillna("N/A").isin(["N/A"])
     blob = df["title"].fillna("") + " " + df["description"].fillna("")
     has_id = blob.str.contains(r"CVE-\d{4}-\d{4,7}|GHSA-", case=False, regex=True)
-    assert bool((has_sev | has_kw | has_stride | has_cwe | has_id).all())
+    # fix-verb × crash-class impact in the title (recall-expansion gate signal)
+    fiximpact = re.compile(
+        r"\b(?:fix|fixes|fixed|prevent|avoid|guard|handle|resolve|correct|patch)\w*"
+        r"\b[^.\n]{0,40}\b(?:crash|panic|segfault|deadlock|hang|freeze|oom"
+        r"|out.of.memory|overflow|underflow|data race|race condition|reorg"
+        r"|non.?determin|infinite loop|use.after.free|null (?:pointer|deref))\b"
+        r"|\b(?:crash|panic|segfault|deadlock|hang|oom|overflow|underflow|reorg"
+        r"|race condition)\b[^.\n]{0,25}"
+        r"\b(?:fix|fixed|prevent|avoid|guard against|resolved|patch)\w*\b", re.I)
+    has_fiximpact = df["title"].fillna("").str.contains(fiximpact)
+    assert bool((has_sev | has_kw | has_stride | has_cwe | has_id | has_fiximpact).all())
 
 
 def test_confidence_values(df):