Skip to content

Commit 04a4579

Browse files
grandchildricegrandchildrice
committed
docs: compare corpus with MineBlockVuln
1 parent 8cb1245 commit 04a4579

9 files changed

Lines changed: 1032 additions & 4 deletions

‎docs/paper/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,8 @@ The working thesis is:
1919
3. CWE comparison: test how much protocol context adds beyond generic weakness
2020
labels when explaining network impact.
2121
4. Prior-work comparison: replicate and extend MineBlockVuln (ESEC/FSE 2022)
22-
across eleven Ethereum clients and six implementation languages.
22+
across eleven Ethereum clients and six implementation languages. See
23+
[`mineblock_replication.md`](mineblock_replication.md).
2324
5. Cross-client recurrence: cluster fixes by specification anchor and measure
2425
whether a fix in one implementation predicts variants in another.
2526

‎docs/paper/mineblock_replication.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
# RQ3 preliminary: replication against MineBlockVuln
2+
3+
## Question
4+
5+
How much of the closest prior blockchain-system vulnerability dataset is
6+
recovered by the current corpus, and what is genuinely new about the
7+
eleven-client dataset?
8+
9+
The comparison target is Yi et al., *An Empirical Study of Blockchain System
10+
Vulnerabilities: Modules, Types, and Patterns* (ESEC/FSE 2022). Its public
11+
SQLite database covers Bitcoin, Ethereum/Geth, Monero, and Stellar.
12+
13+
- Paper: <https://doi.org/10.1145/3540250.3549105>
14+
- Repository: <https://github.com/VPRLab/BlkVulnDataset>
15+
- Public DB link: <https://drive.google.com/file/d/1ntKMt4U4FN6VTi1x-PQw-ZNh1cqKQDLD/view>
16+
- Downloaded DB SHA-256:
17+
`a827279d8abe8b44451df11f46b2be2acc589f75712a75bfff5280580b968525`
18+
19+
The 1.5 GiB external DB is not checked into this repository. Reproduce the
20+
checked-in comparison tables with:
21+
22+
```bash
23+
UV_CACHE_DIR=/tmp/uv-cache uv run python scripts/compare_mineblockvuln.py \
24+
--mineblock-db /path/to/BlkVulnDataset.db
25+
```
26+
27+
## 1. Published populations
28+
29+
| Measure | MineBlockVuln | Current corpus |
30+
|---|---:|---:|
31+
| Scope | 4 blockchain systems | 11 Ethereum clients + consensus specs |
32+
| Ethereum implementation scope | Geth | 5 execution + 6 consensus clients |
33+
| Public Geth vulnerability-issue rows | 367 in DB; 365 final in paper | 407 curated Geth rows |
34+
| Geth Top-20 typed issues | 212 | different protocol/root-cause taxonomy |
35+
| Geth implementation language | Go | 6 languages across all clients |
36+
37+
The current corpus has more curated Geth rows than MineBlockVuln has Geth
38+
issues, but row count alone hides low identity overlap.
39+
40+
## 2. Exact overlap is low
41+
42+
Exact GitHub issue/PR identity:
43+
44+
| Recall stage | Shared legacy refs | Share of 367 |
45+
|---|---:|---:|
46+
| Current raw Geth collection | 88 | 24.0% |
47+
| Current curated Geth corpus | 63 | 17.2% |
48+
| Curated corpus, allowing the Geth ref in any client row | 68 | 18.5% |
49+
50+
The recall decomposition is:
51+
52+
- 63 MineBlockVuln refs appear in the curated Geth corpus;
53+
- 25 additional refs were collected into raw Geth data but removed by the
54+
current gate;
55+
- 279 refs are absent from the raw Geth collection.
56+
57+
Allowing Geth references inherited or cited by other client rows raises raw
58+
coverage only from 88 to 93 refs. The dominant gap is therefore collection,
59+
not the curation gate.
60+
61+
Exact fix-commit identity is lower:
62+
63+
| Measure | Count / share |
64+
|---|---:|
65+
| Unique MineBlockVuln Geth commits linked to vulnerability issues | 1,008 |
66+
| Unique current Geth `fix_commit` values | 337 |
67+
| Shared exact commit SHAs | 65 |
68+
| Legacy commit recall | 6.4% |
69+
| Current Geth commit overlap with legacy | 19.3% |
70+
71+
Commit recall is not directly comparable to issue recall: MineBlockVuln links
72+
multiple commits to one issue, while the current corpus selects one fixing
73+
commit and de-duplicates within a client.
74+
75+
The full row-level decomposition is checked in as
76+
[`tables/mineblock_recall_decomposition.csv`](tables/mineblock_recall_decomposition.csv).
77+
78+
## 3. This falsifies a possible dataset claim
79+
80+
**Do not claim.** The current corpus is not a comprehensive replacement or
81+
superset of MineBlockVuln's historical Geth data.
82+
83+
The raw crawl is trace-biased and misses three quarters of the earlier Geth
84+
issue/PR set by exact URL. The missed set contains not only generic hardening
85+
and operational fixes but also explicitly security-labelled historical issues,
86+
so the gap cannot be dismissed as a pure scope-definition difference.
87+
88+
**Observation.** The new corpus primarily extends the *implementation axis*:
89+
eleven clients, execution and consensus layers, six languages, evidence tiers,
90+
and inline pre/post code. It does not yet establish higher historical recall
91+
within Geth.
92+
93+
**Interpretation.** The strongest novelty claim is cross-implementation and
94+
protocol-native analysis, not “largest” or “most complete” historical
95+
Ethereum-vulnerability coverage.
96+
97+
## 4. Replication of the old type distribution is not yet valid
98+
99+
MineBlockVuln's leading typed Geth categories were:
100+
101+
| Type | Geth issues |
102+
|---|---:|
103+
| Race Condition | 48 |
104+
| Go Panic | 36 |
105+
| Block Related | 21 |
106+
| Check/Validation | 14 |
107+
| Deadlock | 13 |
108+
| Resource Leak | 12 |
109+
| Denial-of-Service | 11 |
110+
| Peer/Node Related | 11 |
111+
112+
Only 45 current rows match a MineBlockVuln Top-20 issue by issue/PR number.
113+
Twenty-three of those matches are in the old “Go Panic” category. This shared
114+
slice is too small and selected to support a distributional replication claim.
115+
116+
The taxonomies also answer different questions:
117+
118+
- MineBlockVuln types mix symptoms (`Go Panic`), root causes (`Race
119+
Condition`), affected objects (`Block Related`), and modules (`RPC Related`);
120+
- the current corpus separates protocol area, root cause, and attack path.
121+
122+
The generated
123+
[`tables/mineblock_type_crosswalk.csv`](tables/mineblock_type_crosswalk.csv)
124+
therefore serves as an annotation queue, not a one-to-one mapping.
125+
126+
## 5. Next validity work
127+
128+
1. Manually stratify the 279 not-collected refs into:
129+
- in-scope vulnerability/security fix;
130+
- security hardening without a concrete vulnerability;
131+
- ordinary reliability bug;
132+
- obsolete Swarm/UI/wallet scope;
133+
- unresolved.
134+
2. Treat the manually reviewed legacy set as an external recall benchmark for
135+
the collection pipeline.
136+
3. Add a MineBlockVuln ingestion source or explicitly narrow the corpus claim to
137+
trace-leaving fixes collected by the documented sources.
138+
4. Compare old and current type distributions only after constructing a
139+
reviewed crosswalk that separates symptom, root cause, module, and trigger.
140+
5. Test the prior paper's language-dependent conclusion (“Go Panic” in
141+
Ethereum) across the six implementation languages. Do not attribute a
142+
Geth/Go observation to the Ethereum protocol.
143+
144+
## 6. Candidate contribution after correction
145+
146+
The defensible replication story is:
147+
148+
> Prior work established that repository mining finds blockchain security work
149+
> beyond CVEs, but studied Ethereum through one Go implementation. The current
150+
> corpus tests which findings survive across eleven implementations and six
151+
> languages, while exposing the recall and taxonomy limits of both datasets.
152+
153+
This is a stronger empirical contribution than repeating the prior paper's
154+
module and type counts on a differently selected sample.

‎docs/paper/snapshot_audit.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,14 @@ All numbers below are generated by
2222
| Fix commit present | 1,959 | 88.0% |
2323
| Post-fix code present | 1,923 | 86.4% |
2424

25-
**Observation.** The repository's current README and manifest agree on 2,225
26-
records. `docs/BUILD_REPORT.md` still describes an earlier 2,333-row build.
27-
Publication tables must not combine those snapshots.
25+
**Observation.** The repository's current README and the top-level
26+
`manifest.n_rows` agree on 2,225 records. However, most nested build
27+
distributions in `data/manifest.json` still sum to 2,333: `by_confidence`,
28+
`by_n_signals`, `by_source`, `by_severity`, and `by_score`.
29+
`docs/BUILD_REPORT.md` also describes that earlier 2,333-row build. Only the
30+
nested authority-tier and severity-source distributions reflect 2,225.
31+
Publication tables must be generated from the Parquet file and must not combine
32+
these snapshots.
2833

2934
## 2. “Silent fix” is definition-sensitive
3035

‎docs/paper/tables/mineblock_overlap_rows.csv‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
ref,mineblock_title,current_rows,current_ids,current_titles,current_root_causes,current_labels
2+
issues/17637,Clef: cliui mixes extraneous and approval-specific Data,1,34e55321e6e9fe6f,"cmd/clef, signer: security fixes (#17554)",missing_input_validation,rpc
3+
issues/2555,geth 1.4.3 panics when calling RPC,1,geth:ethereum-go-ethereum:PR#2575,eth/filter: bugfix which can cause a nil pointer crash when parsing f…,unhandled_error_or_nil,serialization
4+
pull/1189,eth/downloader: handle timeouts more gracefully,1,d42c8037810c51a3,eth/downloader: handle timeouts more gracefully,missing_bounds_check,sync
5+
pull/1314,eth/fetcher: handle and test various DOS attacks,1,geth:ethereum-go-ethereum:PR#1314,eth/fetcher: handle and test various DOS attacks,missing_bounds_check,test
6+
pull/14697,swarm/storage: remove panic on invalid chunk,1,geth:ethereum-go-ethereum:PR#14697,swarm/storage: remove panic on invalid chunk,resource_exhaustion,state-trie
7+
pull/14718,core/vm: fix overflow in gas calculation formula,1,a84406460fceafd5,core/vm: fix overflow in gas calculation formula,integer_overflow_underflow,gas
8+
pull/14790,vendor: update go-stack to fix a sigpanic panic,1,geth:ethereum-go-ethereum:PR#14790,vendor: update go-stack to fix a sigpanic panic,unhandled_error_or_nil,other
9+
pull/1496,"xeth: fix #1485, data race in fiilter creation and event firing",1,bbea6d2b293d94bd,"xeth: fix #1485, data race in fiilter creation and event firing",race_condition,transactions
10+
pull/15665,ethapi: GetTransactionReceipt panic fix,1,geth:ethereum-go-ethereum:PR#15665,ethapi: GetTransactionReceipt panic fix,other,transactions
11+
pull/1568,core: added a running flag to prevent panics in the chainmanager,1,geth:ethereum-go-ethereum:PR#1568,core: added a running flag to prevent panics in the chainmanager,race_condition,block-processing
12+
pull/15714,core: fix panic when stat-ing a tx from a queue-only account,1,geth:ethereum-go-ethereum:PR#15714,core: fix panic when stat-ing a tx from a queue-only account,other,txpool
13+
pull/15794,internal/ethapi: ask transaction pool for pending nonce,1,4e4447c09e1da9e8,internal/ethapi: ask transaction pool for pending nonce,race_condition,transactions
14+
pull/15799,eth: fix tracer panic when running without configs + reexec,1,geth:ethereum-go-ethereum:PR#15799,eth: fix tracer panic when running without configs + reexec,race_condition,metrics-observability
15+
pull/15946,p2p/discv5: fix topic register panic at shutdown,2,ec988c375dbcf16e|geth:ethereum-go-ethereum:PR#15946,p2p/discv5: fix removeTicketRef cached ticket removal | p2p/discv5: fix topic register panic at shutdown,unhandled_error_or_nil,p2p
16+
pull/15995,p2p/discv5: fix removeTicketRef cached ticket removal,1,ec988c375dbcf16e,p2p/discv5: fix removeTicketRef cached ticket removal,unhandled_error_or_nil,p2p
17+
pull/16055,les: fix server panic when discovery disabled,1,geth:ethereum-go-ethereum:PR#16055,les: fix server panic when discovery disabled,other,p2p
18+
pull/16146,Fix race condition in whisperv6/peer.go,1,geth:ethereum-go-ethereum:PR#16146,Fix race condition in whisperv6/peer.go,unhandled_error_or_nil,rlp
19+
pull/16219,whisper: serious security issue fixed,1,geth:ethereum-go-ethereum:PR#16219,whisper: serious security issue fixed,other,metrics-observability
20+
pull/16630,vendor: Fix index out of range panic when size is bigger than 1 TiB,2,f0eea42c13193daa|geth:ethereum-go-ethereum:PR#16630,Geth Titanium (v1.8.7) is a hotfix release to address a leveldb crash affecting archive nodes. | vendor: Fix index out of range panic when size is bigger than 1 TiB,missing_bounds_check|race_condition,evm|state-trie
21+
pull/16891,les: add Skip overflow check to GetBlockHeadersMsg handler,1,2ea9db0643f8dedc,les: add Skip overflow check to GetBlockHeadersMsg handler,integer_overflow_underflow,p2p
22+
pull/17034,les: add announcement safety check to light fetcher,1,3a7f8432f642ca5c,A crash in the LES header fetcher is resolved ([#17034](https://github.com/ethereum/go-ethereum/pull/17034)),other,p2p
23+
pull/17313,swarm/storage/mru: HOTFIX - fix panic in Handler.update,1,geth:ethereum-go-ethereum:PR#17313,swarm/storage/mru: HOTFIX - fix panic in Handler.update,improper_state_update,state-trie
24+
pull/17338,swarm/bmt: ignore data longer then 4096 bytes in Hasher.Write,1,f4339c3f6d541bc0,swarm/bmt: ignore data longer then 4096 bytes in Hasher.Write,missing_input_validation,other
25+
pull/17412,cmd/puppeth: fix nil panic on disconnected stats gathering,3,5803ed65174cc05e|f2cba89aad03bb7b|geth:ethereum-go-ethereum:PR#17412,Fix a crash in the mobile framework when setting a transaction recipient to `nil` ([#17310](https://github.com/ethereum/go-ethereum/pull/17310)). | Fix a transaction chain tracing panic if an invalid parametrization was specified ([#17460](https://github.com/ethereum/go-ethereum/pull/17460)). | cmd/puppeth: fix nil panic on disconnected stats gathering,race_condition|unhandled_error_or_nil,state-trie|transactions
26+
pull/17460,Ensure from < to when tracing chain,2,5803ed65174cc05e|f2cba89aad03bb7b,Fix a crash in the mobile framework when setting a transaction recipient to `nil` ([#17310](https://github.com/ethereum/go-ethereum/pull/17310)). | Fix a transaction chain tracing panic if an invalid parametrization was specified ([#17460](https://github.com/ethereum/go-ethereum/pull/17460)).,race_condition,transactions
27+
pull/17546,"cmd, core, eth, miner, params: configurable gas floor and ceil",1,7d6f43cb4e8fc56a,"cmd, core, eth, miner, params: configurable gas floor and ceil",missing_input_validation,gas
28+
pull/17554,"cmd/clef, signer: security fixes",1,geth:ethereum-go-ethereum:PR#17554,"cmd/clef, signer: security fixes",missing_input_validation,rpc
29+
pull/17639,les: fix distReq.sentChn double close bug,2,69adc7b799e61a35|78526164e8f1b6de,Fix a rare panic in the light client on shutdown ([#17639](https://github.com/ethereum/go-ethereum/pull/17639)). | Fix nil-pointer panic when processing malicious ABIs ([#17653](https://github.com/ethereum/go-ethereum/pull/17653)).,race_condition|unhandled_error_or_nil,sync|transactions
30+
pull/17653,"abi,signer: fix nil dereference in #17633",2,69adc7b799e61a35|78526164e8f1b6de,Fix a rare panic in the light client on shutdown ([#17639](https://github.com/ethereum/go-ethereum/pull/17639)). | Fix nil-pointer panic when processing malicious ABIs ([#17653](https://github.com/ethereum/go-ethereum/pull/17653)).,race_condition|unhandled_error_or_nil,sync|transactions
31+
pull/17734,swarm/network/stream: fix DoS invalid offered hashes length,2,fd11f4cffe069eeb|geth:ethereum-go-ethereum:PR#17734,Fixed panic on ARMl6 arch (64bit struct alignment) ([17766](https://github.com/ethereum/go-ethereum/pull/17766)). | swarm/network/stream: fix DoS invalid offered hashes length,missing_input_validation|resource_exhaustion,database|test
32+
pull/17798,accounts/abi: fix panic in MethodById lookup. Fixes #17797,1,geth:ethereum-go-ethereum:PR#17798,accounts/abi: fix panic in MethodById lookup. Fixes #17797,other,test
33+
pull/18165,trie/database: fix overflow in parent tracking,1,bad60eea57269a6f,trie/database: fix overflow in parent tracking,integer_overflow_underflow,state-trie
34+
pull/18335,downloader: fix edgecase where returned index is OOB for downloader,1,68c6fd5ad6068940,downloader: fix edgecase where returned index is OOB for downloader,missing_bounds_check,sync
35+
pull/18344,cmd/puppeth: fix panic error when export aleth genesis wo/ precompile-addresses,1,geth:ethereum-go-ethereum:PR#18344,cmd/puppeth: fix panic error when export aleth genesis wo/ precompile-addresses,other,precompiles
36+
pull/1843,"eth/downloader: always send termination wakes, clean leftover",1,07afd7e9d31ac311,"eth/downloader: always send termination wakes, clean leftover",race_condition,sync
37+
pull/18487,swarm/pss: fixing race condition,1,geth:ethereum-go-ethereum:PR#18487,swarm/pss: fixing race condition,race_condition,p2p
38+
pull/19262,"cmd, core, eth, trie: get rid of trie cache generations",1,f2c52d056567b78e,"cmd, core, eth, trie: get rid of trie cache generations",resource_exhaustion,sync
39+
pull/19348,core/vm: Correct the Memory Gas Overflow condition,1,fef6b529d2c24653,core/vm: Correct the Memory Gas Overflow condition,integer_overflow_underflow,gas
40+
pull/19401,"eth, les, geth: implement cli-configurable global gas cap for RPC calls",1,501c3b8266695913,"eth, les, geth: implement cli-configurable global gas cap for RPC calls",incorrect_gas_accounting,rpc
41+
pull/1945,Argument parsing can lead to panic in rpc channel,1,geth:ethereum-go-ethereum:PR#1945,Argument parsing can lead to panic in rpc channel,other,rpc
42+
pull/1948,Infinite loop in filters,1,704cf5e79982771c,Infinite loop in filters,integer_overflow_underflow,rpc
43+
pull/195,fix panic on bad tx sender,1,geth:ethereum-go-ethereum:PR#195,fix panic on bad tx sender,serialization_bug,transactions
44+
pull/19562,p2p/discover: fix nil-dereference due to race,1,78dc88cabdf32dd2,p2p/discover: fix nil-dereference due to race,unhandled_error_or_nil,p2p
45+
pull/1966,cmd/geth: fix recover command crash if no param is supplied,1,geth:ethereum-go-ethereum:PR#1966,cmd/geth: fix recover command crash if no param is supplied,other,cli
46+
pull/1974,rpc/api: fix #1972 api regression (nil eth panic) in attach,1,geth:ethereum-go-ethereum:PR#1974,rpc/api: fix #1972 api regression (nil eth panic) in attach,other,rpc
47+
pull/19757,eth: fix sync bloom panic,1,geth:ethereum-go-ethereum:PR#19757,eth: fix sync bloom panic,resource_exhaustion,sync
48+
pull/19764,core: fix fast head updating logic,1,a2dc09bf5b26e0f0,core: fix fast head updating logic,race_condition,build-ci
49+
pull/20013,les: fix panic,1,geth:ethereum-go-ethereum:PR#20013,les: fix panic,unhandled_error_or_nil,p2p
50+
pull/20019,eth: disallow overwrite files via admin.exportChain,1,4b0704f57cd6516e,Fix an import crash caused by a data loss from a previous crash ([#19986](https://github.com/ethereum/go-ethereum/pull/19986)).,consensus_divergence,test
51+
pull/20540,core/state: add more verbosity to panic,1,geth:ethereum-go-ethereum:PR#20540,core/state: add more verbosity to panic,improper_state_update,state-trie
52+
pull/20612,tracers: avoid panic on invalid arguments ,1,geth:ethereum-go-ethereum:PR#20612,tracers: avoid panic on invalid arguments,race_condition,test
53+
pull/2399,eth: fix accidental nil panic on nil errors,1,geth:ethereum-go-ethereum:PR#2399,eth: fix accidental nil panic on nil errors,unhandled_error_or_nil,rpc
54+
pull/2574,eth: skip transaction handling during fast sync,1,03f0d30422e229ba,eth: skip transaction handling during fast sync,consensus_divergence,sync
55+
pull/2711,Fix #2710 Filter race: concurrent map read and map write,1,d783d92d9ff7045b,Fix #2710 Filter race: concurrent map read and map write,race_condition,rpc
56+
pull/3067,"cmd, core, tests: avoid hashing the code in the VM",1,08c1cedc48562681,"cmd, core, tests: avoid hashing the code in the VM",resource_exhaustion,sync
57+
pull/3412,core: bugfix state change race condition in txpool,1,geth:ethereum-go-ethereum:PR#3412,core: bugfix state change race condition in txpool,race_condition,txpool
58+
pull/3519,"les: fixed selectPeer deadlock, improved request distribution",1,870d7159527a8927,"les: fixed selectPeer deadlock, improved request distribution",race_condition,txpool
59+
pull/3591,eth/downloader: fix mutex regression causing panics on fail,1,geth:ethereum-go-ethereum:PR#3591,eth/downloader: fix mutex regression causing panics on fail,missing_input_validation,sync
60+
pull/687,"xeth: fix #640, panic converting nil recipient to hex.",1,geth:ethereum-go-ethereum:PR#687,"xeth: fix #640, panic converting nil recipient to hex.",other,transactions
61+
pull/711,rlp: input limit and other corrections,1,204dd28eca18ee54,rlp: input limit and other corrections,integer_overflow_underflow,transactions
62+
pull/820,rlp: fix list bounds check overflow (found by go-fuzz),1,02b6b0457e294628,rlp: fix list bounds check overflow (found by go-fuzz),missing_bounds_check,rlp
63+
pull/911,"rpc, xeth: fix #881, gracefully handle offline whisper",1,6d5683916ade47e2,"rpc, xeth: fix #881, gracefully handle offline whisper",unhandled_error_or_nil,rpc
64+
pull/954,eth/downloader: fix nil panic caused by wrong variable use,1,geth:ethereum-go-ethereum:PR#954,eth/downloader: fix nil panic caused by wrong variable use,other,sync

0 commit comments

Comments
 (0)