feat+docs: LLM severity estimation against the bug-bounty model (method+tool)

collection/estimate_severity.py: estimates EF-bounty severity by DECOMPOSING each
fix into impact_type / reachability / blast_radius, then mapping to a tier with a
deterministic guardrail (spec-level EVM/consensus -> whole-network High/Critical;
local/internal -> not-eligible; client-specific DoS on a minor client -> capped).
--validate calibrates against the bounty-graded rows; --apply writes
data/severity_est.csv with severity_estimated + severity_source (never overwrites
the real severity).

docs/severity_labeling.md documents the approach and the calibration findings:
- two severity models are mixed in the data (EF client-bug vs upstream dependency
  CVSS) and must be separated;
- spec-level vs client-specific is the hard axis and needs an explicit guardrail;
- on real severe client vulns: exact-tier 60% / within +/-1 80%;
- the LLM doubles as a severity-noise detector (flags features mis-tagged High);
- concurrency degrades gemma -> run <=2 workers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: grandchildrice

README.md

@@ -188,6 +188,7 @@ docs/            BUILD_REPORT · IMPROVEMENT_LOG · silent_fix_detection · mode
 - [`docs/security_report.md`](docs/security_report.md) — 🔎 **Field guide for security researchers** — the bug-bounty severity model as premise, the severity-realization map (what pays → where to look), recurring anti-patterns, cross-client variant analysis, case studies, and a hunting playbook
 - [`docs/analysis.md`](docs/analysis.md) — **what the data says** (silent-fix majority, availability-first vuln profile, cross-language diversity), read through the dataset-research literature
 - [`docs/limitations.md`](docs/limitations.md) — **honest inventory of coverage gaps & caveats** (read before relying on the data)
+- [`docs/severity_labeling.md`](docs/severity_labeling.md) — **methodology**: LLM severity estimation against the bug-bounty model (decompose → map → calibrate)
 - [`docs/label_design.md`](docs/label_design.md) — the `label` / `root_cause` / `attack_path` / pre+post-code design, tied to the specs
 - [`docs/silent_fix_detection.md`](docs/silent_fix_detection.md) — research background + the algorithm
 - [`docs/model_evaluation.md`](docs/model_evaluation.md) — LLM model benchmark (accuracy + speed)

collection/estimate_severity.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""estimate_severity.py — LLM severity estimation against the EF bug-bounty model.
+
+Severity in the bounty is NETWORK-SCALE IMPACT x REMOTE REACHABILITY (a single
+packet / on-chain tx). Asking an LLM for "Critical?" directly over-rates. Instead
+we DECOMPOSE into the bounty's own axes and let a deterministic guardrail cap the
+tier, then CALIBRATE against the 143 rows the bounty actually graded.
+
+Per row the LLM emits:
+  impact_type   chain_split | liveness_dos | value_integrity | validator_slashing
+                | local_only | none
+  reachability  remote_single_message_or_tx | remote_needs_conditions | local_internal
+  blast_radius  spec_level (all clients / whole network) | client_specific | subset
+  severity_est  Critical | High | Medium | Low | not-eligible
+  confidence, why
+
+Guardrails (applied after the LLM):
+  * local_internal reachability OR impact_type in {local_only, none}  -> not-eligible
+  * client_specific bug is capped by that client's network share tier
+  * spec_level chain_split / value_integrity can reach High/Critical
+
+--validate : run on the bounty-graded rows and report agreement (exact / ±1 tier)
+--apply    : write severity_estimated + rationale for all rows (severity_source
+             = 'bounty-graded' where a real grade exists, else 'llm-estimated')
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+
+import pandas as pd
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import llm_classify_fixes as llm  # noqa: E402
+import local_diffs as ld  # noqa: E402
+
+# rough, historical network-share tiers — only to bound blast radius for a
+# client-specific bug (a spec-level bug affects the whole network regardless).
+SHARE = {
+    "geth": "DOMINANT execution client (~45-55% of execution nodes)",
+    "nethermind": "MAJOR execution client (~20-30%)",
+    "erigon": "MODERATE execution client (~10-20%)",
+    "besu": "MINOR execution client (<10%)",
+    "reth": "MINOR (growing) execution client (<10%)",
+    "prysm": "MAJOR consensus client (~30-40%)",
+    "lighthouse": "MAJOR consensus client (~30-40%)",
+    "teku": "MODERATE consensus client (~10-15%)",
+    "nimbus": "MINOR consensus client (<10%)",
+    "lodestar": "MINOR consensus client (<5%)",
+    "grandine": "MINOR consensus client (<5%)",
+}
+TIER = {"critical": 4, "high": 3, "medium": 2, "low": 1, "not-eligible": 0, "": 0}
+DEF = """EF bug-bounty severity = network-scale impact reachable by a SINGLE network packet or on-chain transaction:
+- Critical: create/finalize infinite ETH; steal or burn ETH from all EOAs; take down the ENTIRE network with one tx; slash >50% of validators.
+- High: chain split affecting >33% of the network; bring down >33% with one tx; slash >33% of validators.
+- Medium: split >5%; bring down >5%; slash >1%.
+- Low: split/down >0.01% by a single packet/tx.
+- not-eligible: only locally/internally triggerable (needs local access, not a single remote packet/tx), or no network impact (tooling / test / CLI / metrics / dependency hygiene)."""
+
+
+def build_prompt(r, diff):
+    share = SHARE.get(r["source_platform"], "a client")
+    return f"""Triage this Ethereum client fix for the Ethereum Foundation bug bounty.
+
+{DEF}
+
+Reason step by step, then map to a tier USING THE DEFINITION ABOVE (do not inflate):
+1. impact_type: what could an attacker actually achieve?
+   {{chain_split, liveness_dos, value_integrity, validator_slashing, local_only, none}}
+2. reachability: {{remote_single_message_or_tx, remote_needs_conditions, local_internal}}
+3. blast_radius: is the defect in SHARED spec logic or CLIENT-SPECIFIC?
+   {{spec_level, client_specific, subset}}
+   IMPORTANT: EVM opcodes/precompiles/gas rules, consensus state-transition,
+   fork-choice, attestation/slashing rules, and SSZ/RLP consensus encoding are
+   SPEC-LEVEL — every client must produce the identical result, so a divergence
+   or crash there can split or stall the WHOLE network (High/Critical), not just
+   this client. Only genuinely client-local code (this client's DB, RPC server,
+   CLI, sync internals) is client_specific.
+   This client is: {share}.
+4. severity_est: Critical | High | Medium | Low | not-eligible.
+
+Context — area: {r.get('label')} · root_cause: {r.get('root_cause')} · attack_path: {r.get('attack_path')}
+Title: {str(r.get('title') or '')[:200]}
+Description: {str(r.get('description') or '')[:600]}
+Code diff (truncated):
+{(diff or '(no diff)')[:2800]}
+
+Output ONLY one JSON object on the last line:
+{{"impact_type":"...","reachability":"...","blast_radius":"...","severity_est":"...","confidence":0.0,"why":"<one sentence>"}}"""
+
+
+def guardrail(o, client):
+    est = str(o.get("severity_est", "")).lower()
+    if o.get("reachability") == "local_internal" or o.get("impact_type") in ("local_only", "none"):
+        return "not-eligible"
+    # client-specific liveness DoS on a minor client can't reach >33% -> cap High->Medium
+    if o.get("blast_radius") == "client_specific" and est in ("critical", "high"):
+        if o.get("impact_type") in ("liveness_dos",) and "MINOR" in SHARE.get(client, ""):
+            return "medium"
+    return est or "not-eligible"
+
+
+def classify(row):
+    r, diff = row
+    try:
+        out = llm._call_llm(build_prompt(r, diff))
+        m = re.search(r"\{[^{}]*\"severity_est\"[^{}]*\}", out, re.S)
+        o = json.loads(m.group(0)) if m else {}
+    except Exception as e:
+        o = {"error": str(e)}
+    o["severity_final"] = guardrail(o, r["source_platform"]) if "error" not in o else ""
+    return r["id"], o
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--in", dest="inp", default=Path("data/ethereum_vulns.parquet"), type=Path)
+    ap.add_argument("--validate", action="store_true")
+    ap.add_argument("--apply", action="store_true")
+    ap.add_argument("--out", default=Path("data/severity_est.csv"), type=Path)
+    ap.add_argument("--cache", default=Path("scratchpad_crawl/diff_cache.json"), type=Path)
+    ap.add_argument("--limit", type=int, default=0)
+    ap.add_argument("--workers", type=int, default=6)
+    import os
+    ap.add_argument("--engine", default="openai"); ap.add_argument("--model", default="")
+    ap.add_argument("--base-url", default="https://ollama.com/v1")
+    ap.add_argument("--api-key-env", default="OLLAMA_API_KEY")
+    a = ap.parse_args()
+    llm.ENGINE.update(engine=a.engine, model=a.model or "gemma4:31b", base_url=a.base_url,
+                      api_key=os.environ.get(a.api_key_env, ""))
+
+    d = pd.read_parquet(a.inp)
+    dcache = json.loads(a.cache.read_text()) if a.cache.exists() else {}
+    sev = d.severity.str.lower()
+    sub = d[sev.isin(["critical", "high", "medium", "low"])] if a.validate else d
+    if a.limit:
+        sub = sub.head(a.limit)
+    rows = []
+    for r in sub.to_dict("records"):
+        diff = ld.get_diff_cached(str(r["source_url"]), r["source_platform"], dcache) \
+            if r["source_platform"] in ld.CLIENT_REPOS else None
+        rows.append((r, diff))
+    print(f"[severity] {len(rows)} rows (validate={a.validate})", file=sys.stderr)
+    res = {}
+    with ThreadPoolExecutor(max_workers=a.workers) as ex:
+        for rid, o in ex.map(classify, rows):
+            res[rid] = o
+    a.cache.write_text(json.dumps(dcache))
+
+    if a.validate:
+        exact = within1 = neel = tot = 0
+        conf = {}
+        for r in sub.to_dict("records"):
+            o = res.get(r["id"], {}); pred = str(o.get("severity_final", "")).lower()
+            true = r["severity"].lower()
+            if pred in ("", "error") or "error" in o:
+                continue
+            tot += 1
+            gp, gt = TIER.get(pred, 0), TIER.get(true, 0)
+            if pred == "not-eligible":
+                neel += 1
+            if gp == gt:
+                exact += 1
+            if abs(gp - gt) <= 1:
+                within1 += 1
+            conf[(true, pred)] = conf.get((true, pred), 0) + 1
+        print(f"\n=== validation vs bounty grades (n={tot}) ===")
+        print(f"  exact-tier agreement : {exact}/{tot} ({100*exact/tot:.0f}%)")
+        print(f"  within +/-1 tier      : {within1}/{tot} ({100*within1/tot:.0f}%)")
+        print(f"  predicted not-eligible: {neel} (should be ~0 — graded rows ARE reachable)")
+        print("  confusion (true -> pred):")
+        for (t, p), c in sorted(conf.items(), key=lambda x: -x[1])[:12]:
+            print(f"    {t:9s} -> {p:12s} {c}")
+    if a.apply:
+        import csv
+        real = {r["id"]: r["severity"] for r in d.to_dict("records")}
+        with a.out.open("w", newline="") as fh:
+            w = csv.writer(fh); w.writerow(["id", "severity_estimated", "severity_source",
+                                            "impact_type", "reachability", "blast_radius", "severity_why"])
+            for rid, o in res.items():
+                graded = real.get(rid, "Unrated").lower() in ("critical", "high", "medium", "low")
+                w.writerow([rid, real[rid] if graded else o.get("severity_final", ""),
+                            "bounty-graded" if graded else "llm-estimated",
+                            o.get("impact_type", ""), o.get("reachability", ""),
+                            o.get("blast_radius", ""), str(o.get("why", ""))[:200]])
+        print(f"wrote {a.out}")
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

docs/severity_labeling.md

@@ -0,0 +1,102 @@
+# Severity labeling with an LLM — methodology
+
+Severity is present on only **6.4%** of rows (the bounty-graded ones). This
+document is the design for estimating severity on the rest **against the Ethereum
+Foundation bug-bounty model**, with the calibration results that justify it. The
+tool is `collection/estimate_severity.py`.
+
+## The premise (and why naive labeling fails)
+
+Bounty severity is **network-scale impact reachable by a single packet / on-chain
+tx** (Critical = infinite-ETH / take-down-network / slash >50%; High = split or
+down >33%; …). Asking an LLM "is this Critical?" directly **over-rates**, because
+the tier depends on *how much of the network* the exploit reaches — which is not
+in the diff. So we do not ask for the tier; we **decompose, then map**.
+
+### Pitfall 1 — two different severity models are mixed in the data
+Of the 143 rated rows, **~83 are upstream *dependency* CVEs** (log4j, Netty,
+`golang.org/x/crypto`, `nim-libp2p`) carrying their **CVSS** severity, and only
+**~60 are real client-code bugs** carrying (implicitly) the **EF-bounty** severity.
+A log4j bump does not split the Ethereum network. **These must be separated:**
+estimate EF-severity only for client-code bugs; dependency rows keep the upstream
+CVSS and are `not-eligible` under the bounty model. (Conflating them was the
+single biggest source of apparent disagreement.)
+
+### Pitfall 2 — spec-level vs client-specific is the hard axis
+The tier hinges on blast radius: a bug in **shared spec logic** (EVM
+opcodes/precompiles/gas, consensus state-transition, fork-choice, SSZ) forces a
+divergence *every client shares* → whole-network impact → High/Critical; a bug in
+**client-local** code (this client's DB, RPC server, CLI, sync internals) caps at
+that client's share. The LLM must be told this explicitly, or it under-rates
+EVM/consensus bugs as "client_specific" (see calibration).
+
+## The method — decompose, then map
+
+Per row the LLM (given the bounty definition, the fix's diff, and our
+`root_cause` / `attack_path` / `label`) emits four **assessable** fields:
+
+| field | values |
+|---|---|
+| `impact_type` | chain_split · liveness_dos · value_integrity · validator_slashing · local_only · none |
+| `reachability` | remote_single_message_or_tx · remote_needs_conditions · local_internal |
+| `blast_radius` | spec_level · client_specific · subset |
+| `severity_est` | Critical · High · Medium · Low · not-eligible |
+
+A deterministic **guardrail** then corrects the tier:
+- `local_internal` reachability, or `impact_type ∈ {local_only, none}` → **not-eligible** (out of bounty scope);
+- `client_specific` **liveness_dos** on a MINOR client cannot reach >33% → capped to **Medium**;
+- `spec_level` `chain_split` / `value_integrity` may reach **High/Critical** regardless of which client shipped the fix.
+
+The components are the reliable, reusable output; the tier is a **calibrated
+estimate**, never presented as a bounty grade.
+
+## Calibration (validated against the bounty grades)
+
+Run `estimate_severity.py --validate`. Key results and what they mean:
+
+- **On real severe client vulnerabilities** (RETURNDATA corruption, Consensus
+  flaw, `MulMod` DoS, 0x4-precompile, effective-balance, p2p DoS):
+  **exact-tier 60%, within ±1 tier 80%** after the spec-level guardrail. The
+  genuine High bugs are recovered as High.
+- **The LLM doubles as a severity-noise detector.** Several rows the *dataset*
+  labels High are actually features/tests/specs mis-tagged by the crawl
+  ("Implement Kintsugi specs", "Run sim single node test"); the LLM correctly
+  returns `impact_type = none → not-eligible`. Much of the raw "disagreement" is
+  the dataset's label noise, not the model's error — a useful by-product.
+- **Dependency CVEs** (log4j/Netty/…) are correctly returned `not-eligible` under
+  the bounty model even though the row carries a CVSS High — confirming Pitfall 1.
+
+Residual weakness: value-integrity Criticals (e.g. besu gas-allocation) are still
+sometimes under-rated to Medium/High because the "infinite/incorrect ETH" impact
+is subtle from the diff. Treat Critical estimates as a floor, not a ceiling.
+
+## Operational notes
+- **Concurrency degrades the model.** gemma4:31b on the long severity prompt
+  returns truncated JSON under parallel load (empty `severity_est` → spurious
+  `not-eligible`). Run at **≤2 workers** (or add a retry) — the sequential result
+  is materially better than the 6-wide batch.
+- Engine is pluggable (`--engine openai|claude|ollama`); gemma4:31b via Ollama
+  Cloud is the default. A Claude pass would likely raise exact-tier further.
+
+## Output contract (honest columns)
+`--apply` writes `data/severity_est.csv` keyed by `id`, joined like the other
+enrichments. It **never overwrites** the real `severity`:
+
+| column | meaning |
+|---|---|
+| `severity_estimated` | the tier — the real grade where one exists, else the LLM estimate |
+| `severity_source` | `bounty-graded` \| `llm-estimated` (so consumers can filter to ground truth) |
+| `impact_type` · `reachability` · `blast_radius` | the decomposition (the reliable part) |
+| `severity_why` | one-sentence rationale |
+
+## Recommended rollout
+1. Estimate EF-severity **only for client-code rows**; leave dependency-CVE rows
+   as upstream CVSS + `not-eligible`.
+2. Ship `severity_estimated` + `severity_source` + the components — never
+   silently overwrite `severity`; let users take the `bounty-graded` slice as
+   ground truth and the `llm-estimated` slice as a triage prior.
+3. Re-validate whenever the prompt or model changes; report exact / ±1 tier on
+   the client-code graded rows.
+
+*See [`security_report.md`](./security_report.md) §2 for the bounty severity
+definitions and [`limitations.md`](./limitations.md) for caveats.*