docs: unify canonical dataset metrics

Author: grandchildrice

README.md

@@ -8,8 +8,8 @@ single schema, scored for security relevance, and tiered by evidence strength.
 It is built for training and evaluating spec-compliance / audit tooling:
 > *given the code state just before this fix, would the tool have caught the bug?*
 
-Because Ethereum clients **silently patch** most vulnerabilities (no CVE, vague
-commit message), the hard part is separating real fixes from the flood of
+Because many Ethereum-client fixes lack a recognized advisory ID or a rated
+severity, the hard part is separating real fixes from the flood of
 refactors, dep-bumps and release notes. That separation — **the gate** — is what
 this README explains.
 
@@ -34,8 +34,12 @@ renders as a table (key columns). Full data with inline pre/post code: [`ethereu
 | curated (security-only) | **2,225** |
 | └ essential slice (tier A ∪ B) | **1,808** |
 | by tier | A_authoritative 235 · B_corroborated 1,573 · C_candidate 417 |
-| by confidence | high 337 · medium 1,542 · low 454 |
-| by severity | Critical 3 · High 63 · Medium 60 · Low 21 · Info 853 · Unrated 1,333 |
+| by confidence | high 326 · medium 1,445 · low 454 |
+| by severity | Critical 3 · High 63 · Medium 57 · Low 20 · Info 750 · Unrated 1,332 |
+| public evidence | advisory ID 172 (7.7%) · rated 143 (6.4%) · neither 1,962 (88.2%) |
+
+Canonical definitions and all current counts are fixed in
+[`docs/DATA_SNAPSHOT.md`](docs/DATA_SNAPSHOT.md).
 
 ## How the corpus is built
 
@@ -99,8 +103,10 @@ Pre-gate de-noising, in order (`pipeline/build_security_dataset.py`):
 
 ## Silent-fix detection
 
-Ethereum clients patch ~98–100% of vulnerabilities *silently*. Two research-backed
-methods recover them (full write-up: [`docs/silent_fix_detection.md`](docs/silent_fix_detection.md)):
+The current corpus has 1,962 records (88.2%) with neither a recognized
+CVE/GHSA/RustSec ID nor a rated severity. Two research-backed methods help
+recover low-disclosure fixes (full write-up:
+[`docs/silent_fix_detection.md`](docs/silent_fix_detection.md)):
 
 - **Patch backlinking** — start from a confirmed advisory (OSV/GHSA) and extract
   the exact fixing commit/version (`fix_commit`). Deterministic, high precision.
@@ -128,13 +134,16 @@ The bounty's **scope is exactly this corpus** — the eleven clients plus
 `c-kzg-4844` and the deposit contract. Three consequences shape the `severity`
 column:
 
-- **~94% of rows are `Unrated`.** Most fixes ship with no CVE or advisory, so the
-  bug was never graded. **Unrated ≠ low impact** — the fix record, not the CVE
-  list, is the map (see [`security_report.md`](docs/security_report.md)).
-- **Two severity models coexist.** A minority of rated rows are *upstream
-  dependency CVEs* (log4j, Netty, `golang.org/x/crypto`) carrying **CVSS**
-  severity — a dependency bug does not split the Ethereum network, so it is out of
-  bounty scope. These are kept distinct from real client-bug (EF-bounty) grades.
+- **2,082 rows (93.6%) are `Info` or `Unrated`.** This is a rating-coverage
+  statistic, not an advisory statistic. Under the canonical identifier search,
+  **1,962 rows (88.2%) have neither a recognized advisory ID nor a rating**.
+  **Unrated ≠ low impact** — the fix record, not a CVE score, is the primary
+  evidence (see [`security_report.md`](docs/security_report.md)).
+- **Two severity models coexist.** Of the 143 rated rows, 60 are marked
+  `bounty-graded` and 83 `upstream-cvss`. The latter label is not a verified
+  dependency flag because the current heuristic also catches changelog/release
+  rows; dependency scope requires manual review. Confirmed dependency CVEs carry
+  CVSS and are outside the EF-bounty impact model.
 - **`severity_estimated` (optional LLM pass)** fills the gap by decomposing each
   fix into `impact_type` / `reachability` / `blast_radius` and mapping to the
   bounty tier, calibrated against the graded rows (exact-tier ~60% / ±1 ~80% on

data/manifest.json

@@ -49,50 +49,50 @@
     "security_rows": 2225,
     "low_signal_dropped": 14665,
     "by_confidence": {
-      "medium": 1542,
+      "medium": 1445,
       "low": 454,
-      "high": 337
+      "high": 326
     },
     "by_authority_tier": {
       "B_corroborated": 1573,
       "C_candidate": 417,
       "A_authoritative": 235
     },
     "by_n_signals": {
-      "1": 487,
-      "2": 1154,
-      "3": 566,
+      "1": 462,
+      "2": 1104,
+      "3": 533,
       "4": 120,
       "5": 6
     },
     "by_source": {
-      "geth": 449,
-      "erigon": 446,
-      "nimbus": 272,
-      "lodestar": 233,
-      "lighthouse": 221,
-      "reth": 197,
-      "nethermind": 138,
+      "erigon": 425,
+      "geth": 407,
+      "nimbus": 269,
+      "lodestar": 225,
+      "lighthouse": 217,
+      "reth": 183,
+      "nethermind": 130,
       "teku": 123,
-      "besu": 117,
-      "prysm": 114,
-      "grandine": 21,
+      "prysm": 113,
+      "besu": 112,
+      "grandine": 19,
       "consensus-specs": 2
     },
     "by_severity": {
-      "Unrated": 1333,
-      "Info": 853,
+      "Unrated": 1332,
+      "Info": 750,
       "High": 63,
-      "Medium": 60,
-      "Low": 21,
+      "Medium": 57,
+      "Low": 20,
       "Critical": 3
     },
     "by_score": {
       "0.0": 220,
-      "0.3": 272,
-      "0.5": 1208,
-      "0.8": 296,
-      "0.9": 170,
+      "0.3": 271,
+      "0.5": 1119,
+      "0.8": 289,
+      "0.9": 159,
       "1.0": 167
     },
     "residual_boilerplate_fp": 0,
@@ -106,4 +106,4 @@
     }
   },
   "source": "11 Ethereum execution + consensus clients (past security fixes)"
-}
+}

docs/BUILD_REPORT.md

@@ -4,13 +4,10 @@ Fresh crawl + deterministic curation. Raw snapshot re-collected from all 11
 clients (+ consensus-specs) via `collection/run_pipeline.sh` (uncapped), then the
 curated security-only set derived offline by `pipeline/build_security_dataset.py`.
 
-> **Note:** LLM STRIDE/CWE classification was **skipped** for this build, so
-> `stride=Other` / `cwe_top25=N/A` for every row. The GATE therefore keeps a row
-> only on an *independent non-LLM* signal: a CVE/GHSA id, a rated severity
-> (Critical/High/Medium/Low), or a security-keyword match (`security_score ≥ 0.5`).
-> This yields a smaller, higher-precision set than a classified build — the
-> ~16k rows dropped below are the unrated "stealth" fixes that only an LLM
-> STRIDE label would have admitted.
+> **Current-snapshot rule:** all “After” counts and coverage below are computed
+> from `data/ethereum_vulns.parquet` (2,225 rows). Iteration-local history lives
+> in [`IMPROVEMENT_LOG.md`](./IMPROVEMENT_LOG.md); canonical definitions and
+> hashes live in [`DATA_SNAPSHOT.md`](./DATA_SNAPSHOT.md).
 
 ## Before (raw crawl)
 
@@ -49,44 +46,43 @@ Critical** (geth, besu, teku). Severities preserved through the canonical path.
   across all 11 clients (curated C_candidate + gate-dropped *plausible* rows),
   flagged **696** as real silent fixes. This both promotes classified fixes C→B
   and, via the gate, **admits +453 silent fixes the deterministic keyword gate
-  had dropped** (1,880 → 2,333 rows). Model chosen by an 80-item eval sweep
+  had dropped** in that intermediate iteration. Model chosen by an 80-item eval sweep
   (F1 0.872, precision 0.895); see `docs/model_evaluation.md`. Diffs served
   rate-limit-free by `local_diffs.py` (bulk PR-ref clone). Regenerate via
   `collection/llm_classify_fixes.py --apply` → `data/silent_fix_llm.csv`.
-- by severity: {'Unrated': 963, 'Info': 773, 'High': 63, 'Medium': 54, 'Low': 21, 'Critical': 3}
-  (High/Medium dropped vs iter-1 because T2b removed 49 unrelated CVEs' bogus CVSS severities)
+- by confidence: {'medium': 1445, 'low': 454, 'high': 326}
+- by severity: {'Unrated': 1332, 'Info': 750, 'High': 63, 'Medium': 57, 'Low': 20, 'Critical': 3}
 - by source:
-  - geth: 438
-  - erigon: 371
-  - lodestar: 276
-  - nimbus: 232
-  - lighthouse: 178
-  - reth: 172
-  - prysm: 116
-  - nethermind: 108
-  - besu: 94
-  - teku: 93
-  - grandine: 16
+  - erigon: 425
+  - geth: 407
+  - nimbus: 269
+  - lodestar: 225
+  - lighthouse: 217
+  - reth: 183
+  - nethermind: 130
+  - teku: 123
+  - prysm: 113
+  - besu: 112
+  - grandine: 19
   - consensus-specs: 2
-- security_score distribution: {'0.0': 34, '0.3': 4, '0.5': 1253, '0.8': 423, '0.9': 172, '1.0': 210}
+- security_score distribution: {'0.0': 220, '0.3': 271, '0.5': 1119, '0.8': 289, '0.9': 159, '1.0': 167}
 
 ## Validation checkpoints (issue #89)
 
 - c-kzg-4844 / blst: present (kzg×12, 4844×13, blst×13 in curated)
-- Lodestar: 276 · Nimbus: 232 · Prysm: 116 — all present
+- Lodestar: 225 · Nimbus: 269 · Prysm: 113 — all present
 - `ethereum_specs` source: **0** (spec-divergence crawler returned no matches this run; the 11 clients + consensus-specs are covered)
 
-## Column coverage (n=2,333)
+## Column coverage (n=2,225)
 
 | column | coverage | notes |
 |---|---:|---|
 | `source_url`, `title`, `description`, `attack_path` | 100.0% | attack_path defaults to a best-effort class |
 | `label` (assigned, non-`other`) | **93.4%** | deterministic path/keyword + LLM fallback (`gemma4:31b`) reading the diff or, for no-commit rows, the advisory text |
-| `root_cause` (assigned) | 86.8% | keyword + classifier reason + LLM |
-| `cwe_top25` (from advisory/diff text via LLM) | **24.7%** | read from the link's advisory text / diff even when no fix commit exists — advisories often lack a CWE but the Impact text yields one |
+| `root_cause` (assigned) | 86.5% | keyword + classifier reason + LLM |
+| `cwe_top25` (general CWE label assigned) | **17.8%** | despite the legacy column name, only 130 rows (5.8% overall) are in MITRE's 2025 Top 25 |
 | `fix_commit` / `introduced_in_commit` | **88.0%** | `/commit/` + `/pull/` URLs, GHSA advisory patch-releases, and **inline `#PR` / commit refs parsed from CHANGELOG/release text** (author-linked, high precision) |
-| `pre_fix_code` / `post_fix_code` (inline) | **86.3%** | **98.1% of the 1,959 rows that have a resolved commit** — only 38 committed rows lack a diff (huge/edge-case). The remaining ~266 no-code rows have no single fix commit (advisory/NVD/release) — no code exists to inline. |
-| `silent_fix_prob` (LLM classifier) | 38.5% | classified rows (C_candidate + plausible gate-dropped) |
-| `severity` (bounty-graded, rated) | 6.4% | the EF bug-bounty grades; most fixes are silently patched, unrated |
-| `severity_estimated` (bounty-tier) | **30.3%** | a Low/Medium/High/Critical tier via: 60 `bounty-graded` (ground truth) + 83 `upstream-cvss` (dependency CVEs) + **532 `llm-estimated`** (from the client-code fixes; 1,020 more judged `not-eligible` = not remotely network-impacting). `severity_source` flags provenance. See `docs/severity_labeling.md`. |
-
+| `pre_fix_code` / `post_fix_code` (inline) | **86.4%** | **98.2% of the 1,959 rows with a resolved commit**; 36 committed rows lack post-fix code and 266 have no fix commit |
+| `silent_fix_prob` (LLM classifier) | 40.3% | 897 classified rows (C_candidate + plausible gate-dropped) |
+| `severity` (rated) | 6.4% | 143 rows; provenance is 60 `bounty-graded` and 83 rated `upstream-cvss` |
+| `severity_estimated` (bounty-tier) | **30.3%** | 675 Low/Medium/High/Critical estimates; 1,549 `not-eligible`, one unassessed. See `docs/severity_labeling.md`. |

docs/DATA_SNAPSHOT.md

@@ -0,0 +1,67 @@
+# Canonical dataset snapshot
+
+This page is the single source of truth for counts quoted in current
+documentation. Every value is computed from
+`data/ethereum_vulns.parquet`, not copied from an intermediate build log.
+
+- Rows: **2,225**
+- Parquet SHA-256:
+  `09bb9642023e4fa914268d86e723435cf16626d55f480fc248c43ad6af79e0e7`
+- Recompute: `uv run python scripts/paper_analysis.py`
+- Machine-readable metrics:
+  [`paper/tables/snapshot_metrics.csv`](paper/tables/snapshot_metrics.csv)
+
+## Canonical public-evidence definition
+
+A record is **advisory-linked** when a case-insensitive, syntactically valid
+CVE, GHSA, or RustSec identifier occurs in any of `title`, `description`,
+`issue_id`, `contest`, `source_url`, or `evidence`. A record is **rated** when
+`severity` is Critical, High, Medium, or Low.
+
+| Public evidence | Records | Share |
+|---|---:|---:|
+| Advisory ID and rating | 52 | 2.3% |
+| Advisory ID only | 120 | 5.4% |
+| Rating only | 91 | 4.1% |
+| Neither advisory ID nor rating | **1,962** | **88.2%** |
+| Total | 2,225 | 100.0% |
+
+Thus, 172 records (7.7%) have a recognized advisory ID and 143 (6.4%) have a
+rated severity. Separately, 2,082 (93.6%) are **unrated**. The 93.6% value must
+not be described as “no advisory” or “silent”; it measures rating absence only.
+In current documentation, the precise phrase for 88.2% is “neither a recognized
+advisory ID nor a rated severity.”
+
+The older title/description-only, case-sensitive `CVE-|GHSA-` search finds 109
+records (4.9%). It is retained only as a sensitivity/reproducibility result in
+the paper audit, never as the headline definition.
+
+## Current distributions
+
+| Measure | Count |
+|---|---:|
+| A_authoritative | 235 |
+| B_corroborated | 1,573 |
+| C_candidate | 417 |
+| A ∪ B essential slice | 1,808 |
+| Confidence: high / medium / low | 326 / 1,445 / 454 |
+| Severity: Critical / High / Medium / Low | 3 / 63 / 57 / 20 |
+| Severity: Info / Unrated | 750 / 1,332 |
+| Fix commit present | 1,959 (88.0%) |
+| Post-fix code present | 1,923 (86.4%) |
+| Post-fix code among rows with a fix commit | 1,923 / 1,959 (98.2%) |
+| Fix commit present but post-fix code absent | 36 |
+| CWE assigned (`cwe_top25 != N/A`) | 396 (17.8%) |
+| Assigned CWE in MITRE 2025 Top 25 | 130 (5.8% of all rows) |
+| `silent_fix_prob` assessed | 897 (40.3%) |
+
+Despite its legacy name, `cwe_top25` stores general CWE labels. Only 130 of its
+396 assigned values are members of MITRE's 2025 Top 25.
+
+## Drift policy
+
+`data/manifest.json` distributions must equal counts computed from the Parquet
+snapshot. `tests/test_security_dataset.py` enforces that invariant and also
+checks the four-way advisory/rating partition above. Historical experiment logs
+may contain iteration-local counts, but they must identify themselves as
+historical and link here for the current snapshot.

docs/IMPROVEMENT_LOG.md

@@ -5,6 +5,10 @@ wake-ups; this file is the memory). Each iteration: **acquire additional diff 
 evaluate → improve**, repeated until vulnerability coverage saturates
 (loop-until-dry: N consecutive iterations add no new distinct vulns).
 
+> **Historical ledger:** every count below belongs to the named iteration and
+> is not a current dataset statistic. For the current 2,225-row snapshot, use
+> [`DATA_SNAPSHOT.md`](./DATA_SNAPSHOT.md).
+
 ## Baseline (before loop) — 2026-07-02
 - curated rows: **2096**, rated-severity: **173** (8.3%), high-confidence: **382**
 - Diagnosis: volume-driven (60k+ commit-grep rows), only a tiny authoritative core

docs/analysis.md

@@ -6,8 +6,9 @@ security fixes across the eleven production Ethereum clients.*
 
 > **Key takeaways.** For anyone auditing Ethereum clients (or other blockchain /
 > consensus systems), the data says: (1) the **historical fix record — not the
-> CVE list — is the more complete map**, since ~94% of fixes carry no CVE or
-> advisory; (2) the threat profile is **availability- and consensus-centric**
+> CVE list — is the more complete map**, since 88.2% of records carry neither a
+> recognized advisory ID nor a rated severity; (2) the threat profile is
+> **availability- and consensus-centric**
 > (untrusted network input crashing or diverging a node), not the memory-
 > corruption profile of generic C/C++ datasets; (3) by the bounty's impact
 > definition, **consensus, EVM and crypto code is where an exploitable bug is
@@ -30,23 +31,25 @@ are computed directly over the curated table; fix-size uses the inlined
 post-fix hunks. Method and coverage caveats are in
 [`limitations.md`](./limitations.md).
 
-## 2. Finding 1 — the silent-fix majority
+## 2. Finding 1 — the advisory-light majority
 
 ![Figure 1](figures/fig1_silent_prevalence.png)
 
-**Only 4.9% of curated fixes carry a CVE/GHSA identifier and 6.4% carry any
-rated severity; ≈93.6% are silent** — no advisory, frequently an uninformative
-commit message. This quantifies, for Ethereum clients specifically, the
-silent-patching behaviour that VulFixMiner [Zhou et al., ASE'21] and Sawadogo et
-al. describe qualitatively.
+Under the canonical search across all provenance fields, **172 records (7.7%)
+carry a recognized CVE/GHSA/RustSec identifier**, 143 (6.4%) carry a rated
+severity, and **1,962 (88.2%) carry neither**. The four mutually exclusive
+groups are: ID + rating 52, ID only 120, rating only 91, and neither 1,962.
+The operational definition is fixed in
+[`DATA_SNAPSHOT.md`](./DATA_SNAPSHOT.md).
 
 *Implication.* CVE-anchored datasets — **CVEfixes** [Bhandari et al., 2021] and
 **BigVul** [Fan et al., 2020] — begin from an advisory and walk to the patch, so
-by construction they can only observe the ~5% advisory-linked slice. This corpus
+by construction they can only observe an advisory-linked slice. In this corpus,
+that slice is 7.7% under the canonical identifier definition. This corpus
 is built in the opposite direction (mine the silent majority, then corroborate),
 making it **complementary to, not a subset of**, CVE-anchored resources. A model
-trained solely on CVE-linked fixes never sees the 94% of Ethereum-client fixes
-that never received a CVE.
+trained solely on advisory-linked fixes does not see the 88.2% with neither a
+recognized advisory ID nor a rated severity.
 
 ## 3. Finding 2 — an availability-first, protocol-specific threat profile
 
@@ -171,9 +174,9 @@ Assessed along **Croft et al.**'s [2023] data-quality dimensions:
 - **Currentness.** Freshly crawled (2026), including the newest forks
   (deneb→fulu/gloas, cancun→osaka), where public datasets typically lag years.
 
-The low bars — `severity` (6.4%) and `silent_fix_prob` (40%) — are structural,
-not defects: unrated severity *is* the silent-fix signal (§2), and full-commit
-LLM classification was deliberately bounded (§8).
+The low bars — `severity` (6.4%) and `silent_fix_prob` (40.3%) — describe
+coverage, not ground truth about disclosure. Full-commit LLM classification was
+deliberately bounded (§8).
 
 ## 9. Implications for use