ethereum-vuln-dataset

A curated corpus of past security fixes from the eleven production Ethereum clients (five execution-layer, six consensus-layer). Every row is one historical vulnerability fix — a merged PR, commit, advisory, or CVE — normalized to a single schema, scored for security relevance, and tiered by evidence strength.

It is built for training and evaluating spec-compliance / audit tooling:

given the code state just before this fix, would the tool have caught the bug?

Because many Ethereum-client fixes lack a recognized advisory ID or a rated severity, the hard part is separating real fixes from the flood of refactors, dep-bumps and release notes. That separation — the gate — is what this README explains.

import pandas as pd
df = pd.read_parquet("data/ethereum_vulns.parquet")   # or data/ethereum_vulns.csv

df[df.authority_tier != "C_candidate"]      # the essential slice (1,808 rows)
df[df.confidence == "high"]                 # strongest evidence only

Browse it on GitHub: data/ethereum_vulns.preview.csv renders as a table (key columns). Full data with inline pre/post code: ethereum_vulns.csv or the .parquet.

(A HuggingFace datasets mirror under NyxFoundation/ is planned.)

Dataset at a glance

	rows
raw snapshot (all clients)	18,475
curated (security-only)	2,225
└ essential slice (tier A ∪ B)	1,808
by tier	A_authoritative 235 · B_corroborated 1,573 · C_candidate 417
by confidence	high 326 · medium 1,445 · low 454
by severity	Critical 3 · High 63 · Medium 57 · Low 20 · Info 750 · Unrated 1,332
public evidence	advisory ID 172 (7.7%) · rated 143 (6.4%) · neither 1,962 (88.2%)

Canonical definitions and all current counts are fixed in docs/DATA_SNAPSHOT.md.

How the corpus is built

flowchart TD
    A["11 client repos + spec repos"] -->|"crawl: advisories · stealth PRs · commits ·<br/>releases · CVE / OSV / RustSec / govulncheck"| B["raw snapshot<br/>18,475 rows"]
    B --> N{"de-noise<br/>(deterministic)"}
    N -->|"T1 release-note boilerplate  −11"| D["16,998 rows"]
    N -->|"T2 CI / docs / dep-bump meta-work  −1,417"| D
    N -->|"T2b NVD substring-match false positives  −49"| D
    D --> G[["THE GATE<br/>keep on evidence"]]
    G -->|"no security signal  −14,665"| X["dropped<br/>(low-signal)"]
    G -->|"≥ 1 signal"| K["curated<br/>2,225 rows"]
    K --> T["authority tiers · silent-fix LLM ·<br/>labels · dedup by fix commit"]

Loading

"Gate-dropped" = a row that survived de-noising but for which no independent security signal fired. These 14,665 rows are things like plain refactors, feature PRs, and non-security commits the keyword crawl happened to touch — kept out of the curated set. (A learned LLM pass rescues the ones that are silent fixes; see Silent-fix detection.)

The gate — what decides "in or out"

The gate keeps a row if any one independent signal fires (union = high recall), then tiers it by how much evidence stacked up. No single heuristic is trusted alone.

flowchart LR
    R["a de-noised row"] --> Q{"any independent<br/>security signal?"}
    Q -->|"CVE / GHSA / RustSec id"| K
    Q -->|"rated severity<br/>(Critical…Low)"| K
    Q -->|"security keyword<br/>(score ≥ 0.5)"| K
    Q -->|"fix-verb × crash-class impact<br/>(“fix panic on …”)"| K
    Q -->|"LLM silent-fix classifier<br/>(prob ≥ 0.70)"| K
    Q -->|"STRIDE / CWE label<br/>(if classified)"| K
    Q -->|"none"| X["dropped"]
    K["kept"] --> C{"how strong is<br/>the evidence?"}
    C -->|"advisory id or rated severity"| TA["A_authoritative"]
    C -->|"≥ 2 independent signals"| TB["B_corroborated"]
    C -->|"single signal"| TC["C_candidate"]

Loading

Tier	Meaning	Use it when
A_authoritative	carries an advisory id or an advisory-rated severity — a confirmed vulnerability	you want ground truth
B_corroborated	no id, but ≥ 2 independent signals agree (e.g. strong keyword + sensitive subsystem + LLM says silent-fix)	the default high-precision slice
C_candidate	a single signal fired — broad recall, noisier	you want maximum coverage

The essential slice = A ∪ B. Each row also carries n_signals (how many fired) and confidence (high/medium/low) so you can threshold further.

Pre-gate de-noising, in order (pipeline/build_security_dataset.py):

Stage	Drops	Rationale
T1	release-note / urgency-template rows	a severity from a release header is never trusted (an old build read Nimbus's "critical update required" template as ~95 phantom criticals)
T2	CI / docs / dep-bump meta-work (title-anchored)	not a client-code vulnerability; rows citing an advisory id / strong vuln language / rated severity are protected
T2b	NVD substring-match false positives	crawl_cve matched "geth" inside gethostbyaddr, "Gether Technology", Linux usb: g… — glibc/X.Org/kernel CVEs, not client bugs

Silent-fix detection

The current corpus has 1,962 records (88.2%) with neither a recognized CVE/GHSA/RustSec ID nor a rated severity. Two research-backed methods help recover low-disclosure fixes (full write-up: docs/silent_fix_detection.md):

• Patch backlinking — start from a confirmed advisory (OSV/GHSA) and extract the exact fixing commit/version (fix_commit). Deterministic, high precision.
• Training-free LLM classifier (collection/llm_classify_fixes.py) — an LLM4VFD-style Chain-of-Thought over diff + dev-artifacts + touched subsystem, no fine-tuning. Runs on gemma4:31b (chosen by an 80-item eval sweep: F1 0.872, precision 0.895), or Claude / a local Ollama model. Diffs are served rate-limit-free from bare git clones by collection/local_diffs.py. This pass admitted +453 silent fixes the deterministic gate had missed.

What this corpus does not yet do. Every row reached the classifier because a keyword or an advisory found it first — there are no rows drawn from whole commit histories (contest == "all-commits" is empty), and the curated set covers 0.291% of the 595,966-commit history with a 7.3x per-client spread. The sister project wallet-vuln-dataset moved from this same position to reading histories end to end and measured what the keyword route costs: 81% of the fixes it recovered carry none of the crawler's search terms. Closing that gap here is planned but blocked on the screen, which fails external validation (recall 32.7%, positive rate on random commits 18.7%). The sequence, the pass/fail bar and the measured token cost are in docs/TODO.md.

Severity & scope — the bug-bounty model

Severity here follows the Ethereum Foundation bug bounty, not CVSS. A severity reflects network-scale impact reachable by a single network packet or on-chain transaction:

Tier	Representative definition (reward)
Critical	create/finalize infinite ETH · steal/burn ETH from all EOAs · take down the entire network with one tx · slash >50% of validators (up to $1,000,000)
High	chain split or takedown affecting >33% · slash >33% (up to $50,000)
Medium	>5% split/takedown · slash >1% (up to $10,000)
Low	>0.01% split/takedown by a single packet/tx (up to $2,000)

The bounty's scope is exactly this corpus — the eleven clients plus c-kzg-4844 and the deposit contract. Three consequences shape the severity column:

• 2,082 rows (93.6%) are Info or Unrated. This is a rating-coverage statistic, not an advisory statistic. Under the canonical identifier search, 1,962 rows (88.2%) have neither a recognized advisory ID nor a rating. Unrated ≠ low impact — the fix record, not a CVE score, is the primary evidence (see security_report.md).
• Two severity models coexist. Of the 143 rated rows, 60 are marked bounty-graded and 83 upstream-cvss. The latter label is not a verified dependency flag because the current heuristic also catches changelog/release rows; dependency scope requires manual review. Confirmed dependency CVEs carry CVSS and are outside the EF-bounty impact model.
• bounty-graded in this snapshot is not a published grade. The label was assigned to any rated row the dependency regex missed, so 45 of the 60 rows inherited their tier from a crawler heuristic that rates every cross-client pull request Medium, or High on a ten-keyword match. Only 14 rows carry a maintainer-issued advisory, and the confirmed Critical/High sample is 8, not 18. Later builds split the label into bounty-graded (advisory URL required) and collector-inferred. Full trace: docs/paper/bounty_graded_population_audit.md.
• severity_estimated (optional LLM pass) fills the gap by decomposing each fix into impact_type / reachability / blast_radius and mapping to the bounty tier, calibrated against the graded rows (exact-tier ~60% / ±1 ~80% measured on a hand-picked subset of six real severe bugs — not a population figure; see the audit above). It never overwrites the graded severity; severity_source marks each row bounty-graded · collector-inferred · upstream-cvss · llm-estimated. In the paper analysis, all 110 LLM-generated High labels are conservatively treated as tier-uncertain; the exact-tier evidence is the 8 advisory-confirmed Critical/High records. Method, correction, and calibration: docs/severity_labeling.md.

Clients

Fixes are sourced from each client's own public repository.

Client	Layer	Language	Repository
Geth	execution	Go	ethereum/go-ethereum
Nethermind	execution	C#	NethermindEth/nethermind
Besu	execution	Java	hyperledger/besu
Erigon	execution	Go	erigontech/erigon
Reth	execution	Rust	paradigmxyz/reth
Lighthouse	consensus	Rust	sigp/lighthouse
Lodestar	consensus	TypeScript	ChainSafe/lodestar
Nimbus	consensus	Nim	status-im/nimbus-eth2
Prysm	consensus	Go	prysmaticlabs/prysm
Teku	consensus	Java	Consensys/teku
Grandine	consensus	Rust	grandinetech/grandine

Plus ethereum/consensus-specs and ethereum/execution-specs for spec-divergence fixes.

Schema

Column	Description
id	stable row id
source_platform	client slug (geth, lighthouse, …)
issue_id	PR / issue / commit / advisory id
severity	bug-bounty grade Critical / High / Medium / Low / Info / Unrated (see Severity & scope)
severity_estimated · severity_source	LLM-estimated bounty tier + its provenance (bounty-graded / upstream-cvss / llm-estimated); when the severity pass has run
impact_type · reachability · blast_radius	the severity decomposition (chain_split / liveness_dos / … · remote / local · spec_level / client_specific)
title, description	fix text (verbatim from the client's public repo)
source_url	link to the upstream fix
label	protocol area of the bug (e.g. fork-choice, beacon-chain:attestation, precompiles, blobs) — see docs/label_design.md
root_cause	why it was a bug (missing_bounds_check, integer_overflow_underflow, consensus_divergence, …)
attack_path	how it's triggered (malicious_block, malicious_tx, malicious_p2p_message, …)
pre_fix_code / post_fix_code	inline before/after code as JSON [{file, hunks:[{start_line, code}]}] (multi-file)
files_changed	JSON list of changed paths
fix_commit	fixing commit SHA (/commit/ URL, resolved PR head, or OSV backlink)
introduced_in_commit	parent of the fix commit (last state with the bug present)
security_score	keyword relevance score, 0.0–1.0
silent_fix_prob	learned classifier's p(silent fix), when classified
authority_tier	A_authoritative / B_corroborated / C_candidate
n_signals	number of independent signals that fired
confidence	high / medium / low
stride, cwe_top25	STRIDE / CWE-Top-25 label (optional LLM pass) or Other / N/A

Reproduce

The curated table is derived deterministically from the raw snapshot — no network, no API key:

uv run python pipeline/build_security_dataset.py \
  --in  data/raw/train.classified.parquet \
  --out data/ethereum_vulns.parquet \
  --silent-fix-csv data/silent_fix_llm.csv      # optional: fold in the LLM signal
uv run --with pytest python -m pytest tests/ -q

Re-collecting the raw snapshot (network-bound) or re-running the LLM classification is documented under docs/ and collection/run_pipeline.sh.

Repository layout

data/            ethereum_vulns.parquet (curated) · raw/ · silent_fix_llm.csv · manifest.json
pipeline/        build_security_dataset.py  — deterministic gate + tiering
collection/      crawlers, local_diffs.py, llm_classify_fixes.py, run_pipeline.sh
tests/           quality gates (schema, no-boilerplate, every-row-has-a-signal)
docs/            BUILD_REPORT · IMPROVEMENT_LOG · silent_fix_detection · model_evaluation · collection

Documentation

• docs/security_report.md — 🔎 Auditing Ethereum clients: where the bugs actually live — a field guide for client devs, audit firms, and white-hats: where to look, the six recurring bug patterns, the attack surface, and the cross-implementation variant hunting that turns one client's fix into a lead on another's live bug
• docs/checklist.md — ✅ Security checklist & test plan — per-subsystem review items and unit/e2e tests derived from the actual fixes, ordered by impact
• docs/analysis.md — what the data says (silent-fix majority, availability-first vuln profile, cross-language diversity), read through the dataset-research literature
• docs/limitations.md — honest inventory of coverage gaps & caveats (read before relying on the data)
• docs/severity_labeling.md — methodology: LLM severity estimation against the bug-bounty model (decompose → map → calibrate)
• docs/label_design.md — the label / root_cause / attack_path / pre+post-code design, tied to the specs
• docs/silent_fix_detection.md — research background + the algorithm
• docs/model_evaluation.md — LLM model benchmark (accuracy + speed)
• docs/BUILD_REPORT.md — per-stage before/after for the current snapshot
• docs/IMPROVEMENT_LOG.md — the full iteration ledger
• docs/collection.md — the crawler layer

License

Data: CC-BY-4.0, sourced from each client's own public repository (title / description verbatim from those public sources). Code under collection/ and pipeline/: MIT.