Skip to content

Commit 4f66ab5

Browse files
docs: rewrite the audit guide for its real audience (devs, auditors, white-hats)
Dropped the methodology meta-commentary (circularity, "not a finding") — it serves the writer, not the reader — and rewrote security_report.md as a field guide with a clear arc: the CVE record is empty and the fix history is the real map, here is where the bugs live and what they look like, here is the leverage of eleven implementations of one spec (variant hunting / spec-divergence → chain split), then case studies, lessons, and a playbook. Tightened prose to cut AI-tells (em-dash overuse, "not X but Y", hedging asides). analysis.md §7 similarly trimmed; README severity note and doc pointer updated (removed the dangling §7 reference). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 7f2c4e4 commit 4f66ab5

3 files changed

Lines changed: 217 additions & 224 deletions

File tree

README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -128,9 +128,9 @@ The bounty's **scope is exactly this corpus** — the eleven clients plus
128128
`c-kzg-4844` and the deposit contract. Three consequences shape the `severity`
129129
column:
130130

131-
- **~94% of rows are `Unrated`.** Clients patch silently, so the bug was never
132-
submitted or graded. **Unrated ≠ low impact**see
133-
[`security_report.md`](docs/security_report.md) §7.
131+
- **~94% of rows are `Unrated`.** Most fixes ship with no CVE or advisory, so the
132+
bug was never graded. **Unrated ≠ low impact**the fix record, not the CVE
133+
list, is the map (see [`security_report.md`](docs/security_report.md)).
134134
- **Two severity models coexist.** A minority of rated rows are *upstream
135135
dependency CVEs* (log4j, Netty, `golang.org/x/crypto`) carrying **CVSS**
136136
severity — a dependency bug does not split the Ethereum network, so it is out of
@@ -218,7 +218,7 @@ docs/ BUILD_REPORT · IMPROVEMENT_LOG · silent_fix_detection · mode
218218

219219
## Documentation
220220

221-
- [`docs/security_report.md`](docs/security_report.md) — 🔎 **Audit field guide**where to look (the priority map: which source-code regions concentrate severe bugs), the attack surface, the recurring vulnerability patterns, cross-implementation variant hunting, and lessons transferable to any blockchain / consensus system
221+
- [`docs/security_report.md`](docs/security_report.md) — 🔎 **Auditing Ethereum clients: where the bugs actually live**a field guide for client devs, audit firms, and white-hats: where to look, the six recurring bug patterns, the attack surface, and the cross-implementation variant hunting that turns one client's fix into a lead on another's live bug
222222
- [`docs/analysis.md`](docs/analysis.md)**what the data says** (silent-fix majority, availability-first vuln profile, cross-language diversity), read through the dataset-research literature
223223
- [`docs/limitations.md`](docs/limitations.md)**honest inventory of coverage gaps & caveats** (read before relying on the data)
224224
- [`docs/severity_labeling.md`](docs/severity_labeling.md)**methodology**: LLM severity estimation against the bug-bounty model (decompose → map → calibrate)

docs/analysis.md

Lines changed: 17 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -119,28 +119,23 @@ across implementations. It is the diversity dimension DiverseVul and CrossVul
119119
argue reduces overfitting, obtained here **within a single well-specified
120120
domain**.
121121

122-
## 7. Severity — what the bounty counts, and what we can honestly say
123-
124-
Severity here is the **Ethereum Foundation bug-bounty** grade: an impact reachable
125-
by a single packet / on-chain tx that splits the chain, takes the network down,
126-
corrupts value, or slashes validators (not CVSS). Two honest caveats bound what
127-
this dataset supports:
128-
129-
- **Remote reachability is part of the *definition*, not a finding.** Because a
130-
bug is only graded when it is remotely triggerable, "reachable bugs are the
131-
severe ones" is true *by construction* — not an empirical result. What the data
132-
adds is *where the impactful code is*: consensus / EVM / crypto for
133-
chain-split/value bugs, p2p / sync / RPC for DoS (the audit priority map,
134-
[`security_report.md`](./security_report.md) §2).
135-
- **Only 6.4% of rows were bounty-graded.** The rest are LLM-*estimated*
136-
([`severity_labeling.md`](./severity_labeling.md)), and that estimate is derived
137-
partly from `root_cause` / `attack_path` — so we deliberately draw **no**
138-
"root-cause X raises severity" conclusion from it, which would be circular.
139-
Treat `severity_estimated` as a triage prior and filter to
140-
`severity_source == bounty-graded` for ground truth. Note also that a graded row
141-
reflects **disclosure policy** (which clients publish advisories), not relative
142-
security posture, and fix size does **not** separate severity (median ~51 LOC
143-
high vs 45 overall) — you cannot spot a critical bug by diff size.
122+
## 7. Severity
123+
124+
Severity follows the **Ethereum Foundation bug bounty**, not CVSS: the grade
125+
reflects how much of the network a single packet or transaction could split, halt,
126+
or corrupt. That impact model is what makes crypto, the EVM, and the consensus
127+
state transition the regions to watch for critical bugs, and p2p / sync / RPC the
128+
regions to watch for denial of service (audit guide,
129+
[`security_report.md`](./security_report.md)).
130+
131+
Only 6.4% of rows carry a real bounty grade; the rest are estimated
132+
([`severity_labeling.md`](./severity_labeling.md), about 60% exact and 80% within
133+
one tier on known-severe bugs). Take the graded slice
134+
(`severity_source == bounty-graded`) as ground truth and the estimates as a triage
135+
prior. Two things the graded slice does *not* tell you: it tracks which teams
136+
publish advisories more than which clients are safer, and fix size does not track
137+
severity (median 51 vs 45 LOC), so neither is a shortcut to finding the severe
138+
bugs.
144139

145140
## 8. Data quality and coverage
146141

0 commit comments

Comments
 (0)