|
1 | | -# Dataset analysis |
2 | | - |
3 | | -What this corpus says — read through the lens of the vulnerability-dataset |
4 | | -literature (CVEfixes, BigVul, Devign, CrossVul, DiverseVul, PrimeVul, and Croft |
5 | | -et al.'s data-quality framework). Numbers are for the current snapshot |
6 | | -(n = 2,225 curated rows). |
7 | | - |
8 | | -## Snapshot |
9 | | - |
10 | | -| dimension | value | |
11 | | -|---|---| |
12 | | -| rows | 2,225 (A_authoritative 235 · B_corroborated 1,573 · C_candidate 417) | |
13 | | -| layers | execution 1,259 · consensus 966 | |
14 | | -| languages | Go 945 · Rust 419 · Nim 269 · Java 235 · TypeScript 225 · C# 130 | |
15 | | -| rated severity | **6.4%** · carries a CVE/GHSA id **4.9%** | |
16 | | -| top root causes | missing_input_validation 522 · resource_exhaustion 340 · race_condition 217 · unhandled_error/nil 208 · integer_overflow 185 · consensus_divergence 174 | |
17 | | -| top attack paths | malformed_input 926 · crafted_state 415 · malicious_p2p_message 241 · malicious_attestation 174 | |
18 | | -| fix size | median **45 LOC**, 41% ≤30 LOC · median **2 files**, **43% single-file** | |
19 | | - |
20 | | -## 1. The silent-fix majority is the point (not a bug) |
21 | | - |
22 | | -**Only 6.4% of curated fixes carry a rated severity and 4.9% carry a CVE/GHSA |
23 | | -id.** So **~94% shipped silently** — no advisory, often a vague message. This |
24 | | -quantifies, for Ethereum clients specifically, the phenomenon VulFixMiner and |
25 | | -Sawadogo et al. describe generally. |
26 | | - |
27 | | -*Implication vs prior datasets.* CVE-anchored corpora (**CVEfixes**, **BigVul**) |
28 | | -start from an advisory and walk to the fix, so by construction they can only see |
29 | | -the ~5–6% advisory-linked slice. This corpus is built the other way — surface the |
30 | | -silent majority via multi-signal mining — so it is complementary to, not a subset |
31 | | -of, CVE-anchored datasets. |
32 | | - |
33 | | -## 2. The vulnerability profile is availability-first, and protocol-specific |
34 | | - |
35 | | -Root causes are dominated by **input-validation gaps, resource exhaustion, |
36 | | -races, nil/unhandled errors, integer overflow, and consensus divergence**; |
37 | | -attack paths are dominated by **malformed input, crafted state, and malicious |
38 | | -p2p / attestation messages**. The modal bug is *"untrusted network input crashes |
39 | | -or diverges the node"* — an **availability / consensus** class. |
40 | | - |
41 | | -*Implication.* This differs sharply from the memory-corruption / injection profile |
42 | | -that dominates C/C++ datasets (BigVul, Devign are largely CWE-119/787/476). Two |
43 | | -classes here are essentially **absent from generic datasets**: `consensus_divergence` |
44 | | -(chain split / invalid-block acceptance) and DoS-via-p2p. A detector trained only |
45 | | -on generic CWE data would be blind to the highest-severity Ethereum-specific |
46 | | -class. This argues for domain-specific corpora, echoing CrossVul/DiverseVul's |
47 | | -finding that distribution shift across domains degrades transfer. |
48 | | - |
49 | | -## 3. Fixes are surgical — which supports the counterfactual use-case |
50 | | - |
51 | | -**43% touch a single file, 41% change ≤30 LOC, median 45 LOC.** Security fixes |
52 | | -being small and localized is exactly the prior that VulFixMiner / GraphSPD exploit, |
53 | | -and it matters for this corpus's stated purpose — *"given the pre-fix state, would |
54 | | -the tool have caught it?"*: a tightly-scoped diff + `introduced_in_commit` gives a |
55 | | -clean counterfactual boundary. (The mean of 298 LOC is skewed by a few large |
56 | | -refactor-bundled fixes — median is the honest centre.) |
57 | | - |
58 | | -## 4. Rare axis: one spec, eleven implementations, six languages |
59 | | - |
60 | | -Most vuln datasets are single-language (usually C/C++) and single-project or |
61 | | -project-agnostic. This corpus is **multi-language (6) × multi-implementation (11) |
62 | | -of one protocol**. Because all clients implement the *same* consensus/execution |
63 | | -spec, the **same logical vulnerability can recur across languages** (and the |
64 | | -`label` area is the language-agnostic join key). That enables studies generic |
65 | | -datasets can't support: cross-implementation recurrence, language-specific bug |
66 | | -proneness for an identical spec, and transfer across implementations. This is the |
67 | | -diversity dimension **DiverseVul** and **CrossVul** argue reduces overfitting — |
68 | | -here obtained within a single, well-specified domain. |
69 | | - |
70 | | -## 5. Data quality, by Croft et al.'s dimensions |
71 | | - |
72 | | -- **Accuracy (label correctness).** Multi-signal gate + `authority_tier` + an |
73 | | - LLM classifier validated at ~0.90 precision; labels are *not* human-verified |
74 | | - (see [`limitations.md`](./limitations.md)). The tiering makes the |
75 | | - accuracy/coverage trade-off explicit rather than hidden in a single noisy label |
76 | | - — the direction **PrimeVul** advocates after showing BigVul/Devign labels are |
77 | | - substantially noisy. |
78 | | -- **Uniqueness.** De-duplicated by `fix_commit` within a client (108 removed); |
79 | | - only 2 commits are shared across clients (fork-inherited). Duplication is the |
80 | | - #1 metric-inflation risk **PrimeVul** and **Croft et al.** flag; it is handled. |
81 | | -- **Consistency.** One schema across 11 heterogeneous sources (advisory / stealth |
| 1 | +# A quantitative analysis of the ethereum-vuln-dataset |
| 2 | + |
| 3 | +*Read as a short technical report. Every figure is regenerated from |
| 4 | +`data/ethereum_vulns.parquet` by `scripts/make_figures.py`; n = 2,225 curated |
| 5 | +security fixes across the eleven production Ethereum clients.* |
| 6 | + |
| 7 | +> **Summary.** The corpus is dominated by *silently patched* fixes (≈94% ship |
| 8 | +> with no advisory), its vulnerability profile is **availability- and |
| 9 | +> consensus-centric** rather than the memory-corruption profile of generic C/C++ |
| 10 | +> datasets, its fixes are **localized** (43% single-file), and it spans **six |
| 11 | +> languages implementing one protocol** — a diversity axis absent from prior |
| 12 | +> vulnerability datasets. We interpret each finding against the |
| 13 | +> vulnerability-dataset literature (CVEfixes, BigVul, Devign, CrossVul, |
| 14 | +> DiverseVul, PrimeVul, and Croft et al.'s data-quality framework). |
| 15 | +
|
| 16 | +## 1. Data and method |
| 17 | + |
| 18 | +Each row is one historical vulnerability fix — a merged PR, commit, advisory, or |
| 19 | +CVE — from a client's own public repository, normalized to one schema, tiered by |
| 20 | +evidence strength (`authority_tier`), and labelled with a protocol `area`, |
| 21 | +`root_cause`, `attack_path`, and inline pre-/post-fix code. Distributions below |
| 22 | +are computed directly over the curated table; fix-size uses the inlined |
| 23 | +post-fix hunks. Method and coverage caveats are in |
| 24 | +[`limitations.md`](./limitations.md). |
| 25 | + |
| 26 | +## 2. Finding 1 — the silent-fix majority |
| 27 | + |
| 28 | + |
| 29 | + |
| 30 | +**Only 4.9% of curated fixes carry a CVE/GHSA identifier and 6.4% carry any |
| 31 | +rated severity; ≈93.6% are silent** — no advisory, frequently an uninformative |
| 32 | +commit message. This quantifies, for Ethereum clients specifically, the |
| 33 | +silent-patching behaviour that VulFixMiner [Zhou et al., ASE'21] and Sawadogo et |
| 34 | +al. describe qualitatively. |
| 35 | + |
| 36 | +*Implication.* CVE-anchored datasets — **CVEfixes** [Bhandari et al., 2021] and |
| 37 | +**BigVul** [Fan et al., 2020] — begin from an advisory and walk to the patch, so |
| 38 | +by construction they can only observe the ~5% advisory-linked slice. This corpus |
| 39 | +is built in the opposite direction (mine the silent majority, then corroborate), |
| 40 | +making it **complementary to, not a subset of**, CVE-anchored resources. A model |
| 41 | +trained solely on CVE-linked fixes never sees the 94% of Ethereum-client fixes |
| 42 | +that never received a CVE. |
| 43 | + |
| 44 | +## 3. Finding 2 — an availability-first, protocol-specific threat profile |
| 45 | + |
| 46 | + |
| 47 | + |
| 48 | +Root causes are led by **missing input validation (522)**, **resource exhaustion |
| 49 | +(340)**, **race conditions (217)**, **unhandled error/nil (208)**, **integer |
| 50 | +overflow (185)**, and **consensus divergence (174)**; triggers are led by |
| 51 | +**malformed input (926)**, **crafted state (415)**, and **malicious p2p / |
| 52 | +attestation messages**. The modal defect is *"untrusted network input crashes or |
| 53 | +diverges a node"* — an **availability / consensus** class. |
| 54 | + |
| 55 | +*Implication.* This is a different distribution from the memory-corruption and |
| 56 | +injection classes (CWE-119/787/476/89) that dominate C/C++ corpora such as |
| 57 | +**Devign** [Zhou et al., 2019] and **BigVul**. Two of the highest-severity |
| 58 | +classes here — `consensus_divergence` (chain split / invalid-block acceptance) |
| 59 | +and DoS-via-p2p — are effectively **absent from generic datasets**. A detector |
| 60 | +trained on generic CWE data would be structurally blind to the class that matters |
| 61 | +most for a blockchain client. This is direct evidence for the domain-shift |
| 62 | +concern raised by **CrossVul** [Nikitopoulos et al., 2021] and **DiverseVul** |
| 63 | +[Chen et al., 2023]: cross-domain transfer degrades when the vulnerability |
| 64 | +distribution differs. |
| 65 | + |
| 66 | +## 4. Finding 3 — where the bugs live |
| 67 | + |
| 68 | + |
| 69 | + |
| 70 | +Fixes concentrate in **state/trie**, **p2p networking**, **RPC**, **sync**, and |
| 71 | +the consensus **state-transition** (`beacon-chain:*`, esp. attestation and |
| 72 | +fork-choice). The `label` vocabulary is grounded in the upstream spec repos' |
| 73 | +section names, so it is **language-agnostic**: the same subsystem label applies |
| 74 | +whether the fix landed in Rust (Lighthouse) or Go (Prysm). |
| 75 | + |
| 76 | +*Implication.* Attack surface is dominated by the components that parse |
| 77 | +**untrusted, adversary-controlled data** — the network stack (p2p, RPC, sync) and |
| 78 | +the consensus objects (attestations, blocks). This aligns the empirical bug |
| 79 | +distribution with the threat model and gives auditors a prioritization signal |
| 80 | +that a flat CWE list does not. |
| 81 | + |
| 82 | +## 5. Finding 4 — fixes are localized |
| 83 | + |
| 84 | + |
| 85 | + |
| 86 | +**43% of fixes touch a single file and the median fix changes 45 lines**, though |
| 87 | +a long tail of refactor-bundled fixes pulls the mean to ~300 LOC. Localized |
| 88 | +security patches are exactly the prior that VulFixMiner and **GraphSPD** [Wang et |
| 89 | +al., S&P'23] exploit. |
| 90 | + |
| 91 | +*Implication for the intended use-case.* The corpus is built to answer *"given |
| 92 | +the pre-fix code state, would a tool have caught this?"* A tightly-scoped diff, |
| 93 | +paired with `introduced_in_commit` (the parent commit = last vulnerable state), |
| 94 | +gives a **clean counterfactual boundary** for that evaluation — the localization |
| 95 | +is what makes the pre-/post-fix framing tractable. |
| 96 | + |
| 97 | +## 6. Finding 5 — one protocol, eleven implementations, six languages |
| 98 | + |
| 99 | + |
| 100 | + |
| 101 | +The corpus spans **Go, Rust, Nim, Java, TypeScript, and C#** across 11 clients |
| 102 | +and both layers (execution 1,259 / consensus 966). Crucially, all clients |
| 103 | +implement the **same** consensus/execution specification. |
| 104 | + |
| 105 | +*Implication.* Most vulnerability datasets are single-language (predominantly |
| 106 | +C/C++) and either single-project or project-agnostic. Here, because the |
| 107 | +specification is fixed and the implementations differ, the **same logical |
| 108 | +vulnerability can recur across languages**, joined by the `label` area. This |
| 109 | +enables studies that generic datasets cannot support — cross-implementation |
| 110 | +recurrence, language-specific bug-proneness under an identical spec, and transfer |
| 111 | +across implementations. It is the diversity dimension DiverseVul and CrossVul |
| 112 | +argue reduces overfitting, obtained here **within a single well-specified |
| 113 | +domain**. |
| 114 | + |
| 115 | +## 7. Data quality and coverage |
| 116 | + |
| 117 | + |
| 118 | + |
| 119 | +Assessed along **Croft et al.**'s [2023] data-quality dimensions: |
| 120 | + |
| 121 | +- **Accuracy.** Labels come from spec-grounded rules plus an LLM classifier |
| 122 | + validated at ~0.90 precision; they are *not* human-verified. The |
| 123 | + `authority_tier` / `n_signals` columns expose the accuracy/coverage trade-off |
| 124 | + explicitly, rather than hiding it in a single noisy label — the direction |
| 125 | + **PrimeVul** [Ding et al., 2024] advocates after demonstrating substantial |
| 126 | + label noise in BigVul/Devign. |
| 127 | +- **Uniqueness.** De-duplicated by `fix_commit` within a client (108 rows |
| 128 | + removed); only two commits are shared across clients (fork-inherited). |
| 129 | + Duplication is the leading metric-inflation risk flagged by PrimeVul and Croft |
| 130 | + et al.; it is handled. |
| 131 | +- **Consistency.** One schema over 11 heterogeneous sources (advisory / stealth |
82 | 132 | PR / commit / release / CVE / OSV / RustSec). |
83 | 133 | - **Currentness.** Freshly crawled (2026), including the newest forks |
84 | | - (deneb→fulu/gloas, cancun→osaka) — where most datasets lag years behind. |
85 | | - |
86 | | -## 6. Selection under a <1% base rate |
87 | | - |
88 | | -Security fixes are a fraction of a percent of commits (the "needle in a haystack" |
89 | | -of VulFixMiner). The pipeline responds with a **cheap high-recall pre-filter → |
90 | | -gate → LLM classifier** cascade rather than a blind full-commit scan (measured at |
91 | | -~18 h with precision collapse). The `authority_tier` / `n_signals` columns let a |
92 | | -consumer pick their point on the recall/precision curve — treating selection as a |
93 | | -first-class, tunable step instead of a fixed threshold. |
94 | | - |
95 | | -## 7. It is a *corpus*, not a ready-made benchmark |
96 | | - |
97 | | -**PrimeVul**'s central lesson is that naive splits leak: near-duplicate and |
98 | | -temporally-entangled samples inflate reported model performance. This dataset is a |
99 | | -*corpus* — no train/test split is shipped. A consumer building a benchmark from it |
100 | | -**must** add a temporal and/or by-client split (and treat the fork-shared commits |
101 | | -and the recurring cross-implementation fixes as leakage risks) to get an honest |
102 | | -generalization estimate. |
103 | | - |
104 | | ---- |
105 | | - |
106 | | -*Reproduce these numbers from `data/ethereum_vulns.parquet`; see |
107 | | -[`BUILD_REPORT.md`](./BUILD_REPORT.md) for coverage and [`limitations.md`](./limitations.md) |
108 | | -for caveats.* |
| 134 | + (deneb→fulu/gloas, cancun→osaka), where public datasets typically lag years. |
| 135 | + |
| 136 | +The low bars — `severity` (6.4%) and `silent_fix_prob` (40%) — are structural, |
| 137 | +not defects: unrated severity *is* the silent-fix signal (§2), and full-commit |
| 138 | +LLM classification was deliberately bounded (§8). |
| 139 | + |
| 140 | +## 8. Implications for use |
| 141 | + |
| 142 | +1. **Selection under a <1% base rate.** Security fixes are a fraction of a |
| 143 | + percent of commits — VulFixMiner's "needle in a haystack." The pipeline |
| 144 | + answers with a cheap high-recall pre-filter → gate → LLM cascade rather than a |
| 145 | + blind full-commit scan (measured at ~18 h with precision collapse). Treat |
| 146 | + `authority_tier` as a tunable operating point, not a fixed threshold. |
| 147 | +2. **This is a corpus, not a benchmark.** PrimeVul's central lesson is that naive |
| 148 | + splits leak: near-duplicate and temporally-entangled samples inflate reported |
| 149 | + performance. No train/test split is shipped. A consumer **must** add a |
| 150 | + temporal and/or by-client split — and treat fork-shared commits and recurring |
| 151 | + cross-implementation fixes as leakage risks — to obtain an honest |
| 152 | + generalization estimate. |
| 153 | + |
| 154 | +## References |
| 155 | + |
| 156 | +- Bhandari, Naseer, Moonen. *CVEfixes*. PROMISE 2021. |
| 157 | +- Fan, Li, Wang, Nguyen. *A C/C++ Code Vulnerability Dataset (BigVul)*. MSR 2020. |
| 158 | +- Zhou, Liu, Siow, Du, Liu. *Devign*. NeurIPS 2019. |
| 159 | +- Nikitopoulos et al. *CrossVul*. ESEC/FSE 2021. |
| 160 | +- Chen et al. *DiverseVul*. RAID 2023. |
| 161 | +- Ding et al. *PrimeVul / Vulnerability Detection with Code LMs*. 2024. |
| 162 | +- Croft, Babar, Kholoosi. *Data Quality for ML-based Vulnerability Detection*. ICSE 2023. |
| 163 | +- Zhou et al. *VulFixMiner*. ASE 2021. · Wang et al. *GraphSPD*. IEEE S&P 2023. |
0 commit comments