fix: de-duplicate rows resolving to the same fix commit (within a client)

After #PR resolution, advisory / changelog / release entries collapse onto the
PR/commit they reference — cross_reference ran before that resolution so it
missed them. build_security_dataset now de-dups by (client, fix_commit), keeping
the richest representative (has code > PR/commit URL > tier > text): 108
duplicates removed, 2,333 -> 2,225 rows, essential slice 1,808. README +
BUILD_REPORT numbers and pipeline diagram updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: grandchildrice

README.md

@@ -17,13 +17,12 @@ this README explains.
 import pandas as pd
 df = pd.read_parquet("data/ethereum_vulns.parquet")   # or data/ethereum_vulns.csv
 
-df[df.authority_tier != "C_candidate"]      # the essential slice (1,891 rows)
+df[df.authority_tier != "C_candidate"]      # the essential slice (1,808 rows)
 df[df.confidence == "high"]                 # strongest evidence only
 ```
 
 **Browse it on GitHub:** [`data/ethereum_vulns.preview.csv`](data/ethereum_vulns.preview.csv)
-renders as a table (key columns). Full data: [`ethereum_vulns.csv`](data/ethereum_vulns.csv)
-(2.5 MB) or the `.parquet`.
+renders as a table (key columns). Full data with inline pre/post code: [`ethereum_vulns.csv`](data/ethereum_vulns.csv) or the `.parquet`.
 
 *(A HuggingFace `datasets` mirror under [`NyxFoundation/`](https://huggingface.co/NyxFoundation) is planned.)*
 
@@ -32,9 +31,9 @@ renders as a table (key columns). Full data: [`ethereum_vulns.csv`](data/ethereu
 | | rows |
 |---|---:|
 | raw snapshot (all clients) | 18,475 |
-| curated (security-only) | **2,333** |
-| └ essential slice (tier A ∪ B) | **1,891** |
-| by tier | A_authoritative 239 · B_corroborated 1,652 · C_candidate 442 |
+| curated (security-only) | **2,225** |
+| └ essential slice (tier A ∪ B) | **1,808** |
+| by tier | A_authoritative 235 · B_corroborated 1,573 · C_candidate 417 |
 | by confidence | high 337 · medium 1,542 · low 454 |
 | by severity | Critical 3 · High 63 · Medium 60 · Low 21 · Info 853 · Unrated 1,333 |
 
@@ -49,8 +48,8 @@ flowchart TD
     N -->|"T2b NVD substring-match false positives  −49"| D
     D --> G[["THE GATE<br/>keep on evidence"]]
     G -->|"no security signal  −14,665"| X["dropped<br/>(low-signal)"]
-    G -->|"≥ 1 signal"| K["curated<br/>2,333 rows"]
-    K --> T["authority tiers +<br/>silent-fix LLM enrichment"]
+    G -->|"≥ 1 signal"| K["curated<br/>2,225 rows"]
+    K --> T["authority tiers · silent-fix LLM ·<br/>labels · dedup by fix commit"]
 ```
 
 **"Gate-dropped"** = a row that survived de-noising but for which **no independent

data/ethereum_vulns.csv

@@ -1,6 +1,6 @@
 {
   "domain": "ethereum",
-  "n_rows": 2333,
+  "n_rows": 2225,
   "schema": [
     "id",
     "source_platform",
@@ -40,17 +40,17 @@
     "t2_noise_dropped": 1417,
     "t2b_nvd_fp_dropped": 49,
     "after_t1": 16998,
-    "security_rows": 2333,
+    "security_rows": 2225,
     "low_signal_dropped": 14665,
     "by_confidence": {
       "medium": 1542,
       "low": 454,
       "high": 337
     },
     "by_authority_tier": {
-      "B_corroborated": 1652,
-      "C_candidate": 442,
-      "A_authoritative": 239
+      "B_corroborated": 1573,
+      "C_candidate": 417,
+      "A_authoritative": 235
     },
     "by_n_signals": {
       "1": 487,
@@ -90,7 +90,8 @@
       "1.0": 167
     },
     "residual_boilerplate_fp": 0,
-    "labelled": 2062
+    "labelled": 2062,
+    "deduped_fix_commit": 108
   },
   "source": "11 Ethereum execution + consensus clients (past security fixes)"
 }

data/ethereum_vulns.parquet

@@ -41,10 +41,10 @@ Critical** (geth, besu, teku). Severities preserved through the canonical path.
 
 ## After (curated)
 
-- rows: **2,333**
+- rows: **2,225** (after removing 108 same-fix-commit duplicates)
 - residual boilerplate FP: **0**  ✅
-- **essential slice (A+B): 1,891** (was 173 rated-only) — clean high-precision core
-- by authority_tier: {'B_corroborated': 1652, 'C_candidate': 442, 'A_authoritative': 239}
+- **essential slice (A+B): 1,808** (was 173 rated-only) — clean high-precision core
+- by authority_tier: {'B_corroborated': 1573, 'C_candidate': 417, 'A_authoritative': 235}
 - **learned silent-fix signal (gemma4:31b):** classified **1,519** PR/commit diffs
   across all 11 clients (curated C_candidate + gate-dropped *plausible* rows),
   flagged **696** as real silent fixes. This both promotes classified fixes C→B

data/ethereum_vulns.preview.csv

@@ -425,6 +425,32 @@ def main() -> int:
         report["labelled"] = int((sec["label"].fillna("other") != "other").sum())
         print(f"[labels] joined {report['labelled']} area labels", file=sys.stderr)
 
+        # De-dup rows that resolved to the SAME fix commit within a client:
+        # advisory / changelog / release entries collapse onto the PR/commit they
+        # reference. cross_reference ran before #PR resolution so it missed these.
+        # Keep the richest representative (has code > PR/commit URL > tier > text).
+        fc = sec["fix_commit"].fillna("").astype(str)
+
+        def _dscore(r):
+            s = 8.0 if str(r.get("pre_fix_code") or "[]") != "[]" else 0.0
+            u = str(r.get("source_url") or "")
+            s += 4 if ("/pull/" in u or "/commit/" in u) else 0
+            s += {"A_authoritative": 3, "B_corroborated": 2, "C_candidate": 1}.get(r.get("authority_tier"), 0)
+            s += min(len(str(r.get("description") or "")), 2000) / 1000
+            return s
+
+        sec = sec.assign(_k=sec["source_platform"].astype(str) + "|" + fc, _s=sec.apply(_dscore, axis=1))
+        hasfc = fc.str.len() > 0
+        before = len(sec)
+        dedup = pd.concat([sec[~hasfc],
+                           sec[hasfc].sort_values("_s", ascending=False).drop_duplicates("_k", keep="first")])
+        sec = dedup.drop(columns=["_k", "_s"]).sort_index()
+        report["deduped_fix_commit"] = int(before - len(sec))
+        report["security_rows"] = int(len(sec))
+        report["by_authority_tier"] = {k: int(v) for k, v in sec["authority_tier"].value_counts().items()}
+        print(f"[dedup] removed {report['deduped_fix_commit']} same-fix-commit duplicates "
+              f"-> {len(sec)} rows", file=sys.stderr)
+
     print(json.dumps(report, indent=2, ensure_ascii=False))
     assert report["residual_boilerplate_fp"] == 0, "T1 leak: boilerplate survived into the security set"