Skip to content

Commit 7cf1ea9

Browse files
feat: reduce label 'other' with honest non-protocol labels
Most 'other' rows with a diff were CI/build/test/CLI/metrics changes, not protocol areas. Added build-ci / cli / metrics-observability / test labels (path/keyword rules + LLM vocab) so those are labelled accurately rather than forced into a protocol area. 'other' 251 -> 147 (11.3% -> 6.6%); label coverage 88.7% -> 93.4% (build-ci 103, test 66, cli 42, metrics 21). The remaining 'other' are genuinely generic (event/backend orchestration, advisory-no-diff, vendored). docs/label_design.md + BUILD_REPORT updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 40285ca commit 7cf1ea9

8 files changed

Lines changed: 721 additions & 706 deletions

File tree

data/ethereum_vulns.csv

Lines changed: 232 additions & 232 deletions
Large diffs are not rendered by default.

data/ethereum_vulns.parquet

-206 Bytes
Binary file not shown.

data/ethereum_vulns.preview.csv

Lines changed: 232 additions & 232 deletions
Large diffs are not rendered by default.

data/labels.csv

Lines changed: 237 additions & 237 deletions
Large diffs are not rendered by default.

data/manifest.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@
9090
"1.0": 167
9191
},
9292
"residual_boilerplate_fp": 0,
93-
"labelled": 2078,
93+
"labelled": 2184,
9494
"deduped_fix_commit": 108
9595
},
9696
"source": "11 Ethereum execution + consensus clients (past security fixes)"

docs/BUILD_REPORT.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ Critical** (geth, besu, teku). Severities preserved through the canonical path.
8181
| column | coverage | notes |
8282
|---|---:|---|
8383
| `source_url`, `title`, `description`, `attack_path` | 100.0% | attack_path defaults to a best-effort class |
84-
| `label` (assigned, non-`other`) | **88.4%** | deterministic path/keyword + LLM fallback (`gemma4:31b`) reading the diff or, for no-commit rows, the advisory text |
84+
| `label` (assigned, non-`other`) | **93.4%** | deterministic path/keyword + LLM fallback (`gemma4:31b`) reading the diff or, for no-commit rows, the advisory text |
8585
| `root_cause` (assigned) | 86.8% | keyword + classifier reason + LLM |
8686
| `cwe_top25` (from advisory/diff text via LLM) | **24.7%** | read from the link's advisory text / diff even when no fix commit exists — advisories often lack a CWE but the Impact text yields one |
8787
| `fix_commit` / `introduced_in_commit` | **88.0%** | `/commit/` + `/pull/` URLs, GHSA advisory patch-releases, and **inline `#PR` / commit refs parsed from CHANGELOG/release text** (author-linked, high precision) |

docs/label_design.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,14 @@ pyspec `process_*` operations:
8585

8686
### Cross-cutting (either layer)
8787
`crypto` (hashing, secp256k1, BLS/KZG math) · `serialization` (SSZ / RLP not tied
88-
to one area) · `database` (storage / DB layer) · `other` (keep rare; forces review).
88+
to one area) · `database` (storage / DB layer) · `build-ci` (build system, CI,
89+
Docker, dependency manifests, vendored deps) · `cli` (command-line / node startup
90+
/ flags) · `metrics-observability` (metrics, diagnostics, tracing, logging) ·
91+
`test` (test-only changes) · `other` (keep rare; forces review).
92+
93+
The non-protocol labels (`build-ci` / `cli` / `metrics-observability` / `test`)
94+
exist so a CI/build/test/CLI change is labelled *accurately* rather than forced
95+
into a protocol area — they keep the `other` bucket small and honest.
8996

9097
---
9198

pipeline/enrich_labels.py

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -176,10 +176,17 @@ def resolve_advisory(client, ghsa, summary):
176176
(r"devp2p|/p2p|discover|/eth/protocol|/eth/handler|snap[-_]?protocol|wire", "p2p"),
177177
(r"block[-_]?process|/core/blockchain|verifyheader|process(?:block|_block)|state_transition", "block-processing"),
178178
]
179-
_X = [ # cross-cutting (checked last)
179+
_X = [ # cross-cutting (checked after protocol rules; most-informative first)
180180
(r"crypto|secp256|ecrecover|keccak|\bhash\b|blst|bn256|bls12|schnorr", "crypto"),
181181
(r"\bssz\b|serial|encode|decode|marshal|unmarshal|codec", "serialization"),
182182
(r"leveldb|rocksdb|pebble|/db\b|database|/ethdb|/storage/kv", "database"),
183+
# non-protocol but real areas (keeps 'other' honest instead of forced)
184+
(r"\.github|\.circleci|dockerfile|docker-compose|\.gradle\b|gradle/|makefile\b"
185+
r"|/build/|verification-metadata|renovate|/vendor/|docs/vulnerab|go\.mod\b|go\.sum"
186+
r"|package-lock|yarn\.lock|Cargo\.(?:toml|lock)|\.ya?ml\b", "build-ci"),
187+
(r"metric|diagnostic|prometheus|grafana|observ|telemetr|tracing|\botel\b", "metrics-observability"),
188+
(r"\bcmd/|/cli/|main\.(?:go|rs)\b|BesuCommand|/flags?/|command\.java", "cli"),
189+
(r"_test\.(?:go|rs|py|ts|js)|/tests?/|testhelper|mock_|spec\.(?:ts|js)|\bfuzz", "test"),
183190
]
184191
_C = [(re.compile(p, re.I), l) for p, l in _C]
185192
_E = [(re.compile(p, re.I), l) for p, l in _E]
@@ -299,7 +306,8 @@ def _group(hunks):
299306
"evm", "opcodes", "precompiles", "gas", "transactions", "txpool",
300307
"block-processing", "state-trie", "rlp", "p2p", "sync", "engine-api",
301308
"blobs", "eof", "rpc"]
302-
CROSS = ["crypto", "serialization", "database", "other"]
309+
CROSS = ["crypto", "serialization", "database", "build-ci", "cli",
310+
"metrics-observability", "test", "other"]
303311
RC_ENUM = [v for _, v in _RC] + ["improper_state_update", "other"]
304312
AP_ENUM = [v for _, v in _AP] + ["internal_only"]
305313

0 commit comments

Comments
 (0)