Skip to content

Commit 7abf961

Browse files
grandchildricegrandchildrice
committed
docs: compare CWE with protocol context
1 parent aa4618f commit 7abf961

8 files changed

Lines changed: 451 additions & 2 deletions

‎docs/paper/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,9 @@ The working thesis is:
2525
EF-bounty `severity_estimated` population without mixing upstream CVSS. Its
2626
primary exact-tier result uses 60 bounty grades; 110 original LLM High
2727
labels are retained as a traceable `tier-uncertain` candidate queue.
28-
5. CWE comparison: test how much protocol context adds beyond generic weakness
29-
labels when explaining network impact.
28+
5. [`cwe_context_comparison.md`](cwe_context_comparison.md) measures how much
29+
Ethereum root-cause and protocol-location context remains when generic CWE
30+
metadata is absent.
3031
6. Prior-work comparison: replicate and extend MineBlockVuln (ESEC/FSE 2022)
3132
across eleven Ethereum clients and six implementation languages. See
3233
[`mineblock_replication.md`](mineblock_replication.md).
@@ -40,6 +41,7 @@ From the repository root:
4041
```bash
4142
UV_CACHE_DIR=/tmp/uv-cache uv run python scripts/paper_analysis.py
4243
UV_CACHE_DIR=/tmp/uv-cache uv run python scripts/severity_analysis.py
44+
UV_CACHE_DIR=/tmp/uv-cache uv run python scripts/cwe_context_analysis.py
4345
git diff --exit-code docs/paper/tables
4446
```
4547

‎docs/paper/cwe_context_comparison.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# RQ3: what Ethereum context adds beyond CWE categories
2+
3+
## Question
4+
5+
How much of the dataset—and especially its confirmed severe slice—is described
6+
by generic CWE labels, and what information remains available when CWE is
7+
absent?
8+
9+
## 1. Coverage is sparse and provenance-dependent
10+
11+
Across the frozen 2,225-record snapshot:
12+
13+
| Representation | Records | Coverage |
14+
|---|---:|---:|
15+
| Any assigned CWE | 396 | 17.8% |
16+
| Member of MITRE 2025 CWE Top 25 | 130 | 5.8% |
17+
| Non-`other` Ethereum root cause | 1,925 | 86.5% |
18+
| Non-`other` protocol/subsystem label | 2,078 | 93.4% |
19+
20+
The legacy column name `cwe_top25` is misleading: it stores 57 distinct CWE
21+
identifiers, only some of which belong to MITRE's 2025 Top 25.
22+
23+
CWE coverage must not be interpreted as the prevalence of CWE-applicable
24+
defects. It is strongly associated with public advisory provenance. The
25+
completed advisory review found an assigned CWE on 26/36 direct-client advisory
26+
records (72.2%), compared with 326/2,053 records without a recognized advisory
27+
ID (15.9%). This is annotation and disclosure selection as well as vulnerability
28+
semantics.
29+
30+
## 2. Confirmed severe vulnerabilities are poorly represented
31+
32+
Among the 18 bounty-graded Critical/High records:
33+
34+
- 3/18 (16.7%) have any CWE (`CWE-190`, `CWE-400`, and `CWE-439`);
35+
- 0/18 are members of MITRE's 2025 CWE Top 25;
36+
- 17/18 (94.4%) have a non-`other` root cause;
37+
- 17/18 (94.4%) have a non-`other` protocol/subsystem label;
38+
- of the 15 records without a CWE, 14 (93.3%) still have both a root cause and
39+
a protocol label.
40+
41+
The zero Top-25 count is a descriptive result from a small sample, not proof
42+
that the Top 25 is irrelevant to Ethereum. With n=18, the paper should not make
43+
a population-level coverage claim from this slice alone. It does show that a
44+
Top-25-only empirical design would omit every confirmed severe example in this
45+
snapshot.
46+
47+
## 3. CWE and protocol context answer different questions
48+
49+
CWE generally describes a software weakness mechanism. The dataset's
50+
`root_cause` normalizes that mechanism for client code, while `label` locates it
51+
in an Ethereum protocol or implementation surface.
52+
53+
This distinction is visible even for common CWE values:
54+
55+
| CWE | Records | Distinct root causes | Distinct protocol labels |
56+
|---|---:|---:|---:|
57+
| CWE-248 | 60 | 2 | 16 |
58+
| CWE-20 | 50 | 4 | 12 |
59+
| CWE-362 | 48 | 1 | 14 |
60+
| CWE-400 | 42 | 2 | 12 |
61+
| CWE-190 | 22 | 1 | 12 |
62+
63+
For example, `CWE-190` says integer overflow/underflow, but its 22 records span
64+
12 protocol/subsystem labels. The protocol label distinguishes whether the
65+
arithmetic defect appears in gas accounting, opcodes, fork choice, state
66+
transition, or another surface. These are not interchangeable categories.
67+
68+
Conversely, Ethereum-specific failure classes are often missing CWE:
69+
70+
| Root cause | Records | CWE assigned | Coverage |
71+
|---|---:|---:|---:|
72+
| `consensus_divergence` | 174 | 2 | 1.1% |
73+
| `incorrect_gas_accounting` | 48 | 0 | 0.0% |
74+
| `serialization_bug` | 19 | 0 | 0.0% |
75+
| `integer_overflow_underflow` | 185 | 24 | 13.0% |
76+
| `missing_input_validation` | 522 | 61 | 11.7% |
77+
78+
This does not mean CWE cannot represent these rows. It means the current public
79+
and generated metadata rarely provides that representation, while the
80+
Ethereum-specific axes remain available.
81+
82+
## 4. Quantified complementarity
83+
84+
Of the 1,829 records without an assigned CWE:
85+
86+
- 1,544 (84.4%) still have a non-`other` root cause;
87+
- 1,745 (95.4%) still have a non-`other` protocol/subsystem label;
88+
- 1,541 (84.3%) have both.
89+
90+
The contribution is therefore not a replacement taxonomy. It is a
91+
two-coordinate description:
92+
93+
> generic weakness mechanism × Ethereum protocol location.
94+
95+
That representation retains structured information for 1,541 records that
96+
would otherwise be uncategorized in a CWE-only analysis.
97+
98+
## 5. Defensible paper claim
99+
100+
> CWE metadata is sparse and disclosure-biased in this corpus: only 396/2,225
101+
> records have any CWE, and none of the 18 confirmed Critical/High records maps
102+
> to the MITRE 2025 Top 25. Ethereum-specific root-cause and protocol-location
103+
> axes provide both coordinates for 1,541/1,829 records lacking CWE, exposing
104+
> consensus divergence, gas accounting, and other protocol failure modes that a
105+
> CWE-only dataset view does not operationally capture.
106+
107+
The phrase “does not operationally capture” is important. The data supports a
108+
metadata-coverage claim, not the stronger ontological claim that CWE is
109+
incapable of representing blockchain-client defects.
110+
111+
## Generated evidence
112+
113+
- [`tables/cwe_context_coverage.csv`](tables/cwe_context_coverage.csv)
114+
- [`tables/cwe_root_cause_coverage.csv`](tables/cwe_root_cause_coverage.csv)
115+
- [`tables/cwe_semantic_multiplicity.csv`](tables/cwe_semantic_multiplicity.csv)
116+
- [`tables/bounty_severe_cwe_audit.csv`](tables/bounty_severe_cwe_audit.csv)
117+
- [`tables/reviewed_scope_cwe_coverage.csv`](tables/reviewed_scope_cwe_coverage.csv)

‎docs/paper/tables/bounty_severe_cwe_audit.csv‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
id,source_platform,severity_estimated,cwe_top25,cwe_known,cwe_2025_top25,root_cause,label,attack_path,title,source_url
2+
besu:ghsa-advisory:GHSA-4456-w38r-m53x,besu,Critical,CWE-190,True,False,integer_overflow_underflow,gas,malicious_tx,Gas allocation error in CALL operations in Besu EVM [GHSA-4456-w38r-m53x CVE-2022-36025],https://github.com/besu-eth/besu/security/advisories/GHSA-4456-w38r-m53x
3+
geth:ghsa-advisory:GHSA-m6gx-rhvj-fh52,geth,Critical,CWE-400,True,False,resource_exhaustion,other,malformed_input,Denial of service due to Go CVE-2020-28362 [GHSA-m6gx-rhvj-fh52 CVE-2020-28362],https://github.com/ethereum/go-ethereum/security/advisories/GHSA-m6gx-rhvj-fh52
4+
besu:hyperledger-besu:GHSA-7pg2-p5vj-xp5h,besu,High,N/A,False,False,missing_input_validation,evm,malformed_input,"SHL, SHR, and SAR operations trigger native exception at key values",https://github.com/besu-eth/besu/security/advisories/GHSA-7pg2-p5vj-xp5h
5+
475682d10d661b75,consensus-specs,High,N/A,False,False,other,p2p,internal_only,Ethereum 2.0 Networking Specification,https://github.com/ethereum/consensus-specs/pull/1328
6+
geth:ethereum-go-ethereum:GHSA-2gjw-fg97-vg3r,geth,High,N/A,False,False,resource_exhaustion,crypto,malicious_p2p_message,DoS via malicious p2p message,https://github.com/ethereum/go-ethereum/security/advisories/GHSA-2gjw-fg97-vg3r
7+
geth:ethereum-go-ethereum:GHSA-69v6-xc2j-r2jf,geth,High,CWE-439,True,False,improper_state_update,precompiles,malicious_tx,Shallow copy in the 0x4 precompile could lead to EVM memory corruption,https://github.com/ethereum/go-ethereum/security/advisories/GHSA-69v6-xc2j-r2jf
8+
geth:ethereum-go-ethereum:GHSA-9856-9gg9-qcmq,geth,High,N/A,False,False,consensus_divergence,opcodes,malicious_tx,RETURNDATA corruption via datacopy,https://github.com/ethereum/go-ethereum/security/advisories/GHSA-9856-9gg9-qcmq
9+
geth:ethereum-go-ethereum:GHSA-jm5c-rv3w-w83m,geth,High,N/A,False,False,integer_overflow_underflow,opcodes,malformed_input,Denial of service via `MulMod`,https://github.com/ethereum/go-ethereum/security/advisories/GHSA-jm5c-rv3w-w83m
10+
geth:ethereum-go-ethereum:GHSA-xw37-57qp-9mm4,geth,High,N/A,False,False,consensus_divergence,transactions,crafted_state,Consensus flaw during block processing,https://github.com/ethereum/go-ethereum/security/advisories/GHSA-xw37-57qp-9mm4
11+
ece24a10f23119be,lighthouse,High,N/A,False,False,missing_input_validation,beacon-chain:attestation,malicious_attestation,Improve validator monitor experience for high validator counts (#3728),https://github.com/sigp/lighthouse/commit/830efdb5c2723209030e83378b2d8e44f9a6b97f
12+
lighthouse:sigp-lighthouse:GHSA-wm9c-xvqq-5c28,lighthouse,High,N/A,False,False,missing_input_validation,beacon-chain:sync-committee,crafted_state,Incorrect processing of effective balances in Electra epoch processing,https://github.com/sigp/lighthouse/security/advisories/GHSA-wm9c-xvqq-5c28
13+
3f495d3767bd5a5f,lodestar,High,N/A,False,False,missing_input_validation,beacon-chain:execution-payload,crafted_state,Run sim single node test with Geth catalyst to finality,https://github.com/ChainSafe/lodestar/pull/3287
14+
4f09bedbf1707694,lodestar,High,N/A,False,False,missing_input_validation,fork-choice,malicious_attestation,Implement Kintsugi specs :tea: (the Merge November sprint PR),https://github.com/ChainSafe/lodestar/pull/3418
15+
7482623489bc990c,lodestar,High,N/A,False,False,missing_input_validation,beacon-chain:justification-and-finality,malicious_p2p_message,feat: eip-8025 optional execution proofs,https://github.com/ChainSafe/lodestar/pull/8918
16+
dc7e3fa111ce0e2a,lodestar,High,N/A,False,False,consensus_divergence,beacon-chain:execution-payload,crafted_state,rename random to prevRandao as per the kiln v2 specs,https://github.com/ChainSafe/lodestar/pull/3787
17+
b89b66ccf953baca,nimbus,High,N/A,False,False,serialization_bug,beacon-chain:attestation,malicious_attestation,2 changes getting us closer to interop,https://github.com/status-im/nimbus-eth2/pull/918
18+
b1a6e88b2f095f12,reth,High,N/A,False,False,missing_input_validation,rpc,malformed_input,feat(trie): geth-compatible zero hashes for non-existent accounts in eth_getProof,https://github.com/paradigmxyz/reth/pull/24360
19+
f960a5728e79e59b,reth,High,N/A,False,False,consensus_divergence,engine-api,malformed_input,feat: Check CL/Reth capability compatibility,https://github.com/paradigmxyz/reth/pull/20348

‎docs/paper/tables/cwe_context_coverage.csv‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
population,metric,rows,denominator,percent
2+
all_snapshot,cwe_known,396,2225,17.798
3+
all_snapshot,cwe_2025_top25,130,2225,5.843
4+
all_snapshot,root_cause_known,1925,2225,86.517
5+
all_snapshot,protocol_label_known,2078,2225,93.393
6+
all_snapshot,no_cwe_but_root_cause_known,1544,1829,84.418
7+
all_snapshot,no_cwe_but_protocol_label_known,1745,1829,95.407
8+
all_snapshot,no_cwe_but_both_context_axes_known,1541,1829,84.254
9+
authority_A_or_B,cwe_known,307,1808,16.98
10+
authority_A_or_B,cwe_2025_top25,104,1808,5.752
11+
authority_A_or_B,root_cause_known,1628,1808,90.044
12+
authority_A_or_B,protocol_label_known,1742,1808,96.35
13+
authority_A_or_B,no_cwe_but_root_cause_known,1327,1501,88.408
14+
authority_A_or_B,no_cwe_but_protocol_label_known,1476,1501,98.334
15+
authority_A_or_B,no_cwe_but_both_context_axes_known,1325,1501,88.274
16+
bounty_graded,cwe_known,8,60,13.333
17+
bounty_graded,cwe_2025_top25,3,60,5.0
18+
bounty_graded,root_cause_known,47,60,78.333
19+
bounty_graded,protocol_label_known,55,60,91.667
20+
bounty_graded,no_cwe_but_root_cause_known,39,52,75.0
21+
bounty_graded,no_cwe_but_protocol_label_known,49,52,94.231
22+
bounty_graded,no_cwe_but_both_context_axes_known,39,52,75.0
23+
confirmed_bounty_critical_or_high,cwe_known,3,18,16.667
24+
confirmed_bounty_critical_or_high,cwe_2025_top25,0,18,0.0
25+
confirmed_bounty_critical_or_high,root_cause_known,17,18,94.444
26+
confirmed_bounty_critical_or_high,protocol_label_known,17,18,94.444
27+
confirmed_bounty_critical_or_high,no_cwe_but_root_cause_known,14,15,93.333
28+
confirmed_bounty_critical_or_high,no_cwe_but_protocol_label_known,15,15,100.0
29+
confirmed_bounty_critical_or_high,no_cwe_but_both_context_axes_known,14,15,93.333
30+
llm_estimated,cwe_known,186,1552,11.985
31+
llm_estimated,cwe_2025_top25,68,1552,4.381
32+
llm_estimated,root_cause_known,1411,1552,90.915
33+
llm_estimated,protocol_label_known,1490,1552,96.005
34+
llm_estimated,no_cwe_but_root_cause_known,1232,1366,90.19
35+
llm_estimated,no_cwe_but_protocol_label_known,1341,1366,98.17
36+
llm_estimated,no_cwe_but_both_context_axes_known,1230,1366,90.044

‎docs/paper/tables/cwe_root_cause_coverage.csv‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
root_cause,rows,cwe_known_rows,cwe_known_percent,cwe_2025_top25_rows,cwe_2025_top25_percent,distinct_cwe_labels
2+
missing_input_validation,522,61,11.686,47,9.004,13
3+
resource_exhaustion,340,91,26.765,21,6.176,6
4+
other,300,15,5.0,4,1.333,14
5+
race_condition,217,58,26.728,1,0.461,7
6+
unhandled_error_or_nil,208,110,52.885,39,18.75,14
7+
integer_overflow_underflow,185,24,12.973,0,0.0,2
8+
consensus_divergence,174,2,1.149,1,0.575,2
9+
improper_state_update,90,6,6.667,1,1.111,6
10+
missing_bounds_check,90,22,24.444,15,16.667,7
11+
incorrect_gas_accounting,48,0,0.0,0,0.0,0
12+
crypto_misuse,31,6,19.355,1,3.226,3
13+
serialization_bug,19,0,0.0,0,0.0,0
14+
reentrancy,1,1,100.0,0,0.0,1

‎docs/paper/tables/cwe_semantic_multiplicity.csv‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
cwe,rows,in_2025_top25,distinct_root_causes,distinct_protocol_labels,leading_root_cause,leading_root_cause_rows
2+
CWE-248,60,False,2,16,unhandled_error_or_nil,59
3+
CWE-20,50,True,4,12,missing_input_validation,39
4+
CWE-362,48,False,1,14,race_condition,48
5+
CWE-400,42,False,2,12,resource_exhaustion,41
6+
CWE-476,33,True,3,10,unhandled_error_or_nil,29
7+
CWE-190,22,False,1,12,integer_overflow_underflow,22
8+
CWE-770,22,True,2,9,resource_exhaustion,21
9+
CWE-401,20,False,1,7,resource_exhaustion,20
10+
CWE-125,13,True,1,6,missing_bounds_check,13
11+
CWE-287,6,False,2,3,missing_input_validation,5
12+
CWE-667,6,False,3,5,race_condition,4
13+
CWE-119,5,False,3,3,missing_bounds_check,3
14+
CWE-327,4,False,1,2,crypto_misuse,4
15+
CWE-674,4,False,1,4,resource_exhaustion,4
16+
CWE-772,4,False,1,3,resource_exhaustion,4
17+
CWE-1284,3,False,2,3,missing_input_validation,2
18+
CWE-191,3,False,2,3,integer_overflow_underflow,2
19+
CWE-284,3,True,2,3,other,2
20+
CWE-502,3,True,2,1,missing_input_validation,2
21+
CWE-129,2,False,1,2,missing_bounds_check,2
22+
CWE-22,2,True,1,1,missing_input_validation,2
23+
CWE-24,2,False,1,1,unhandled_error_or_nil,2
24+
CWE-285,2,False,1,2,missing_input_validation,2
25+
CWE-703,2,False,1,2,unhandled_error_or_nil,2
26+
CWE-704,2,False,2,2,unhandled_error_or_nil,1
27+
CWE-833,2,False,1,2,race_condition,2
28+
CWE-1104,1,False,1,1,other,1
29+
CWE-116,1,False,1,1,missing_input_validation,1
30+
CWE-1395,1,False,1,1,other,1
31+
CWE-193,1,False,1,1,missing_bounds_check,1
32+
CWE-200,1,True,1,1,other,1
33+
CWE-214,1,False,1,1,other,1
34+
CWE-250,1,False,1,1,other,1
35+
CWE-276,1,False,1,1,missing_input_validation,1
36+
CWE-295,1,False,1,1,missing_input_validation,1
37+
CWE-306,1,True,1,1,missing_input_validation,1
38+
CWE-323,1,False,1,1,crypto_misuse,1
39+
CWE-366,1,False,1,1,race_condition,1
40+
CWE-369,1,False,1,1,unhandled_error_or_nil,1
41+
CWE-396,1,False,1,1,missing_input_validation,1
42+
CWE-404,1,False,1,1,unhandled_error_or_nil,1
43+
CWE-416,1,True,1,1,unhandled_error_or_nil,1
44+
CWE-436,1,False,1,1,improper_state_update,1
45+
CWE-437,1,False,1,1,improper_state_update,1
46+
CWE-439,1,False,1,1,improper_state_update,1
47+
CWE-449,1,False,1,1,resource_exhaustion,1
48+
CWE-494,1,False,1,1,unhandled_error_or_nil,1
49+
CWE-662,1,False,1,1,race_condition,1
50+
CWE-672,1,False,1,1,reentrancy,1
51+
CWE-682,1,False,1,1,other,1
52+
CWE-754,1,False,1,1,unhandled_error_or_nil,1
53+
CWE-762,1,False,1,1,improper_state_update,1
54+
CWE-778,1,False,1,1,unhandled_error_or_nil,1
55+
CWE-787,1,True,1,1,missing_bounds_check,1
56+
CWE-788,1,False,1,1,other,1
57+
CWE-789,1,False,1,1,other,1
58+
CWE-838,1,False,1,1,unhandled_error_or_nil,1

0 commit comments

Comments
 (0)